韩友洪

韩友洪的博客

他的个人主页  他的博客

使用httpsqs出现原因不明错误

韩友洪  2010年04月09日 星期五 16:57 | 2403次浏览 | 0条评论

httpsqs


 在一个简单的项目中使用了httpsqs

流程非常简单:

1,python脚本定时扫描日志-》排重-》写入httpsqs队列

2,python daemon读日志,取出数据,处理

最初工作状态还不错,正常工作了2个星期

但是清明自己休息了5天,回来公司后发现数据不正确

[root@rrd ~]# curl "http://127.0.0.1:1218/?name=target_adv&opt=status"
HTTP Simple Queue Service v1.2
------------------------------
Queue Name: target_adv
Maximum number of queues: 100000000
Put position of queue (2nd lap): 9814174
Get position of queue (1st lap): 14309449
Number of unread queue: 95504725

[root@rrd ~]#curl "http://127.0.0.1:1218/?name=target_adv&opt=get"
HTTPSQS_GET_END

出现错误:

1,无法读取数据!

2,Number of unread queue: 95504725数值大了一个数量级

3,现在把以前的HTTPSQS停掉,换上新端口,仍然无法读取,但是可以查看!

root@rrd mailapp]# curl "http://127.0.0.1:1300/?name=target_adv&opt=get"
HTTPSQS_GET_END

[root@rrd mailapp]# curl "http://127.0.0.1:1300/?name=target_adv&opt=view&pos=1"
yansss8@sina.com,2010-03-30 11:45:32,中国,北京,北京,1214900334

4, 使用python接口直接调原始文件,正常

     from tokyocabinet import btree
        db = btree.BTree()
        f="/data2/quene2/httpsqs.db"
        db.open(f,btree.BDBOWRITER | btree.BDBOCREAT)
        cur = db.cursor()
        cur.first()
    
        emails = {}
      
        while True:
            data = cur.rec()
            #key = data[0]
            line = data[1].strip()
            o = line.split(",")
            email = o[0]
            ts = o[1]
          
            #self.compareTs(ts,ts)
            if emails.has_key(email):
                try:
                    pre = emails[email]
                    pre_ts = pre.split(",")[1]
                  
                    #当前时间更加新
                    if self.compareTs(ts,pre_ts):
                        emails[email] = line
                except:
                    print "split error"
                  
            else:
               emails[email] = line
           # db.out(key)
         
           # time.sleep(1)
            #print line  
            try:  
                cur.next()
            except:
                print "NO record"
                break
                time.sleep(1)
      
        for  line in emails.values():
            self.deal(line)

评论

我的评论:

发表评论

请 登录 后发表评论。还没有在Zeuux哲思注册吗?现在 注册 !

暂时没有评论

Zeuux © 2024

京ICP备05028076号