Mercurial > kiritsu

annotate FeedSources.py @ 8:4d61c56473c2 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed a problem where some feeds would have unpickleable expatreaders.
author Atul Varma <varmaa@toolness.com>
date Fri, 18 Apr 2008 17:13:02 -0700
parents 56bd30b89166
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
1 import RssAtomFeed
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
2 import ImapFeed
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
3
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
4 FEED_SOURCES = [RssAtomFeed, ImapFeed]
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
5
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
6 FEED_INFO = {}
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
7
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
8 def _filenameForFeed( name ):
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
9 # TODO: This should be more robust and deal with weird characters
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
10 # better. It should also be a one-to-one mapping from feed names
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
11 # to filenames.
5
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
12 return "feed-%s.rss" % name
4
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
13
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
14 for feedSource in FEED_SOURCES:
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
15 for feed in feedSource.getFeedInfo():
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
16 key = feed["name"]
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
17 feed["filename"] = _filenameForFeed( key )
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
18 feed["source"] = feedSource
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
19 assert key not in FEED_INFO
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
20 FEED_INFO[key] = feed
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
21 del feed
df9d4e704257 Made the architecture a little cleaner, but it's still pretty messy; I really need to turn some OO-like-hacks into actual OO constructions. At least the Config.py structure is simpler.
Atul Varma <varmaa@toolness.com>
parents:
diff changeset
22 del feedSource
5
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
23
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
24 if __name__ == "__main__":
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
25 import sys
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
26 name = sys.argv[1]
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
27 feed = FEED_INFO[name]
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
28 source = feed["source"]
56bd30b89166 A subprocess pool is now used to parallelize the fetching of feeds, which makes the update process much faster.
Atul Varma <varmaa@toolness.com>
parents: 4
diff changeset
29 source.update( feed )