CollecTor Filesystem Protocol

CollecTor Filesystem Protocol.

class bushel.collector.filesystem.CollecTorIndexCompression(extension: str, decompress: Callable[[bytes], bytes])

Enumeration of supported compression types for CollecTor indexes.

Name	Description
UNCOMPRESSED	Uncompressed
BZ2	bzip2
XZ	xz
GZ	gzip

Variables:extension (str) – Filename extension with leading dot (“.”).

class bushel.collector.filesystem.CollectorOutBridgeDescsMarker

Enumeration of marker names under the “bridge-descriptors” directory as specified in §5.2 of [collector-protocol].

Name	Description
EXTRA_INFO	Bridge extra-info descriptors (§5.2.1)
SERVER_DESCRIPTOR	Bridge server descriptors (§5.2.1)
STATUS	Bridge statuses (§5.2.2)

class bushel.collector.filesystem.CollectorOutRelayDescsMarker

Enumeration of marker names under the “relay-descriptors” directory as specified in §5.3 of [collector-protocol].

Name	Description
CONSENSUS	Network status consensuses (§5.3.2)
EXTRA_INFO	Relay extra-info descriptors (§5.3.2)
SERVER_DESCRIPTOR	Relay server descriptors (§5.3.2)
VOTE	Network status votes (§5.3.2)

class bushel.collector.filesystem.CollectorOutSubdirectory

Enumeration of subdirectory names under the “out” directory as specified in §5.0 of [collector-protocol].

Name	Description
BRIDGE_DESCRIPTORS	Bridge descriptors (§5.2)
EXIT_LISTS	Exit lists (§5.1)
RELAY_DESCRIPTORS	Relay descriptors (§5.3)
TORPERF	Torperf and Onionperf (§5.1)
WEBSTATS	Web server access logs (§5.4)

class bushel.collector.filesystem.CollectorRecentSubdirectory

Enumeration of subdirectory names under the “recent” directory as specified in §4.0 of [collector-protocol].

Name	Description
BRIDGE_DESCRIPTORS	Bridge descriptors (§4.2)
EXIT_LISTS	Exit lists (§4.1.1)
RELAY_DESCRIPTORS	Relay descriptors (§4.3)
TORPERF	Torperf and Onionperf (§4.1.2)
WEBSTATS	Web server access logs (§4.4)

bushel.collector.filesystem.collector_422_filename(valid_after: datetime.datetime, fingerprint: str) → str

Create a filename for a bridge status according to §4.2.2 of the [collector-protocol]. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> fingerprint = "BA44A889E64B93FAA2B114E02C2A279A8555C533" # Serge
>>> collector_422_filename(valid_after, fingerprint)
'20181119-150000-BA44A889E64B93FAA2B114E02C2A279A8555C533'

Parameters:

• valid_after (datetime) – The valid-after time.
• fingerprint (str) – The fingerprint of the bridge authority.

Returns:

Filename as a str.

bushel.collector.filesystem.collector_431_filename(valid_after: datetime.datetime) → str

Create a filename for a network status consensus according to §4.3.1 of the [collector-protocol]. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_431_filename(valid_after)
'2018-11-19-15-00-00-consensus'

Parameters:valid_after (datetime) – The valid-after time. Returns:Filename as a str.

bushel.collector.filesystem.collector_433_filename(valid_after: datetime.datetime, v3ident: str, digest: str) → str

Create a filename for a network status vote according to §4.3.3 of the [collector-protocol].

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> v3ident = "D586D18309DED4CD6D57C18FDB97EFA96D330566"  # moria1
>>> digest = "663B503182575D242B9D8A67334365FF8ECB53BB"
>>> collector_433_filename(valid_after, v3ident, digest)  # doctest: +ELLIPSIS
'2018-11-19-15-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-663B...3BB'

Paths in the Collector File Structure Protocol using this filename expect upper-case hex-encoded SHA-1 digests.

>>> v3ident = "d586d18309ded4cd6d57c18fdb97efa96d330566"  # Lower case gets corrected
>>> digest = "663b503182575d242b9d8a67334365ff8ecb53bb"  # Lower case gets corrected
>>> collector_433_filename(valid_after, v3ident, digest)  # doctest: +ELLIPSIS
'2018-11-19-15-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-663B...3BB'

Parameters:

• valid_after (datetime) – The valid-after time.
• v3ident (str) – The v3ident of the directory authority.
• digest (str) – The digest of the vote.

Returns:

Filename as a str.

bushel.collector.filesystem.collector_434_filename(valid_after: datetime.datetime) → str

Create a filename for a microdesc-flavoured network status consensus according to §4.3.4 of the [collector-protocol]. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_434_filename(valid_after)
'2018-11-19-15-00-00-consensus-microdesc'

Parameters:valid_after (datetime) – The valid-after time. Returns:Filename as a str.

bushel.collector.filesystem.collector_521_path(subdirectory: bushel.collector.filesystem.CollectorOutSubdirectory, marker: Union[bushel.collector.filesystem.CollectorOutRelayDescsMarker, bushel.collector.filesystem.CollectorOutBridgeDescsMarker], published: datetime.datetime, digest: str) → str

Create a path according to §5.2.1 of the [collector-protocol]. This is used for server-descriptors and extra-info descriptors for both relays and bridges. For example:

>>> subdirectory = CollectorOutSubdirectory.RELAY_DESCRIPTORS
>>> marker = CollectorOutRelayDescsMarker.SERVER_DESCRIPTOR
>>> published = datetime.datetime(2018, 11, 19, 9, 17, 56)
>>> digest = "a94a07b201598d847105ae5fcd5bc3ab10124389"
>>> collector_521_path(subdirectory, marker, published, digest)  # doctest: +ELLIPSIS
'relay-descriptors/server-descriptor/2018/11/a/9/a94a...389'

Paths in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-1 digests.

>>> digest = "A94A07B201598D847105AE5FCD5BC3AB10124389" # Upper case gets corrected
>>> collector_521_path(subdirectory, marker, published, digest)  # doctest: +ELLIPSIS
'relay-descriptors/server-descriptor/2018/11/a/9/a94a...389'

Parameters:

• subdirectory (str) – The subdirectory under the “out” directory to use. Standard values can be found in CollectorOutSubdirectory.
• marker (str) – The marker under the subdirectory to use. Standard values can be found in CollectorOutRelayDescsMarker and CollectorOutBridgeDescsMarker.
• published (datetime) – The published time.
• digest (str) – The hex-encoded SHA-1 digest for the descriptor. The case will automatically be fixed to lower-case.

Returns:

Path for the descriptor as a str.

bushel.collector.filesystem.collector_521_substructure(published: datetime.datetime, digest: str) → str

Create a path substructure according to §5.2.1 of the [collector-protocol]. This is used for server-descriptors and extra-info descriptors for both relays and bridges. For example:

>>> published = datetime.datetime(2018, 11, 19, 9, 17, 56)
>>> digest = "a94a07b201598d847105ae5fcd5bc3ab10124389"
>>> collector_521_substructure(published, digest)
'2018/11/a/9'

Paths in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-1 digests.

>>> digest = "A94A07B201598D847105AE5FCD5BC3AB10124389" # Upper case gets corrected
>>> collector_521_substructure(published, digest)
'2018/11/a/9'

Parameters:

• published (datetime) – The published time.
• digest (str) – The hex-encoded SHA-1 digest for the descriptor. The case will automatically be fixed to lower-case.

Returns:

Path substructure as a str.

bushel.collector.filesystem.collector_522_path(subdirectory: bushel.collector.filesystem.CollectorOutSubdirectory, marker: Union[bushel.collector.filesystem.CollectorOutRelayDescsMarker, bushel.collector.filesystem.CollectorOutBridgeDescsMarker], valid_after: datetime.datetime, filename: str) → str

Create a path according to §5.2.2 of the [collector-protocol]. This is used for bridge statuses, and network-status consensuses (both ns- and microdesc- flavors) and votes. For a bridge status for example:

>>> subdirectory = CollectorOutSubdirectory.BRIDGE_DESCRIPTORS
>>> marker = CollectorOutBridgeDescsMarker.STATUSES
>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> fingerprint = "BA44A889E64B93FAA2B114E02C2A279A8555C533" # Serge
>>> filename = collector_422_filename(valid_after, fingerprint)
>>> collector_522_path(subdirectory, marker, valid_after, filename)  # doctest: +ELLIPSIS
'bridge-descriptors/statuses/2018/11/19/20181119-150000-BA44...533'

Or alternatively for a network-status consensus:

>>> subdirectory = CollectorOutSubdirectory.RELAY_DESCRIPTORS
>>> marker = CollectorOutRelayDescsMarker.CONSENSUS
>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> filename = collector_431_filename(valid_after)
>>> collector_522_path(subdirectory, marker, valid_after, filename)
'relay-descriptors/consensus/2018/11/19/2018-11-19-15-00-00-consensus'

Parameters:

• subdirectory (str) – The subdirectory under the “out” directory to use. Standard values can be found in CollectorOutSubdirectory.
• marker (str) – The marker under the subdirectory to use. Standard values can be found in CollectorOutRelayDescsMarker and CollectorOutBridgeDescsMarker.
• valid_after (datetime) – The valid_after time.
• filename (str) – The filename to use as a str, typically created with collector_422_filename() for bridge statuses, collector_431_filename() for network-status consensuses, or collector_433_filename() for network-status votes.

Returns:

Path for the descriptor as a str.

bushel.collector.filesystem.collector_522_substructure(valid_after: datetime.datetime) → str

Create a path substructure according to §5.2.2 of the [collector-protocol]. This is used for bridge statuses, and network-status consensuses and votes. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_522_substructure(valid_after)
'2018/11/19'

Parameters:valid_after (datetime) – The valid-after time. Returns:Path substructure as a str.

bushel.collector.filesystem.collector_533_substructure(valid_after: datetime.datetime) → str

Create a substructure according to §5.3.3 of the [collector-protocol]. This is used for microdesc-flavored consensuses and microdescriptors. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_533_substructure(valid_after)
'2018/11'

bushel.collector.filesystem.collector_534_consensus_path(valid_after)

Create a path according to §5.3.4 of the [collector-protocol] for a microdesc-flavored consensus. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> collector_534_consensus_path(valid_after)  # doctest: +ELLIPSIS
'relay-descriptors/microdesc/2018/11/consensus-microdesc/19/2018-11-1...sc'

bushel.collector.filesystem.collector_534_microdescriptor_path(valid_after: datetime.datetime, digest: str) → str

Create a path according to §5.3.4 of the [collector-protocol] for a microdescriptor. For example:

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> digest = "00d91cf96321fbd536dd07e297a5e1b7e6961ddd10facdd719716e351453168f"
>>> collector_534_microdescriptor_path(valid_after, digest)  # doctest: +ELLIPSIS
'relay-descriptors/microdesc/2018/11/micro/0/0/00d...e351453168f'

This path in the Collector File Structure Protocol using this substructure expect lower-case hex-encoded SHA-256 digests.

>>> valid_after = datetime.datetime(2018, 11, 19, 15)
>>> digest = "00D91CF96321FBD536DD07E297A5E1B7E6961DDD10FACDD719716E351453168F"
>>> collector_534_microdescriptor_path(valid_after, digest)  # doctest: +ELLIPSIS
'relay-descriptors/microdesc/2018/11/micro/0/0/00d...e351453168f'

bushel.collector.filesystem.collector_index_path(compression: bushel.collector.filesystem.CollecTorIndexCompression) → str

Create a path to the CollecTor index file, using the specified compression algorithm.

Parameters:compression (CollecTorIndexCompression) – Compression algorithm to use.