ploneintranet.microblog package

Submodules

ploneintranet.microblog.interfaces module

interface ploneintranet.microblog.interfaces.IMicroblogContext

Bases: plone.uuid.interfaces.IUUIDAware

Marker interface for non-SiteRoot objects with a local microblog. Such objects should be adaptable to provide a UUID.

interface ploneintranet.microblog.interfaces.IMicroblogTool

Bases: ploneintranet.microblog.interfaces.IStatusContainer

Provide IStatusContainer as a site utility.

is_most_recent_in_thread(statusupdate)

Returns True if this statusupdate is the most recent reply in its thread, or if it’s a toplevel post without replies.

Returns False if there are more recent statusupdates in this thread.

get()

Query for an attribute description

keys(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

context_values(context, min=None, max=None, limit=100)

Filter IStatusUpdate values by IMicroblogContext object. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

thread_keys(thread_id, min=None, max=None, limit=100)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns ids with the same toplevel thread parent (including parent)

context_items(context, min=None, max=None, limit=100)

Filter (key, IStatusUpdate) items by IMicroblogContext object. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items context <object> filters on StatusUpdates keyed to that context’s UUID.

iteritems(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

context_keys(context, min=None, max=None, limit=100)

Filter IStatusUpdate keys by IMicroblogContext object. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

iterkeys(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

mention_keys(mentions, min=None, max=None, limit=100)

Filter IStatusUpdate keys by mentions. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

mention_values(mentions, min=None, max=None, limit=100)

Filter IStatusUpdate values by mentions. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

thread_values(thread_id, min=None, max=None, limit=100)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns updates with the same toplevel thread parent (including parent)

clear()

Empty the status storage and all indexes.

add(status)

Add a IStatusUpdate.

Actual storage may be queued for later insertion by the implementation.

Returns 1 on completion of synchronous insertion. Returns 0 when the actual insertion is queued for later processing.

thread_items(thread_id, min=None, max=None, limit=100)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns items with the same toplevel thread parent (including parent)

user_items(users, min=None, max=None, limit=100)

Filter (key, IStatusUpdate) items by iterable of userids. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

values(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

user_values(users, min=None, max=None, limit=100)

Filter IStatusUpdate values by iterable of userids. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

itervalues(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

items(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

user_keys(users, min=None, max=None, limit=100)

Filter IStatusUpdate keys by iterable of userids. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

mention_items(mentions, min=None, max=None, limit=100)

Filter (key, IStatusUpdate) items by mentions. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

interface ploneintranet.microblog.interfaces.IStatusContainer

Bases: zope.interface.Interface

Manages read/write access to, and storage of, IStatusUpdate instances.

IStatusContainer provides a subset of a ZODB IBTree interface.

Some IBTree methods are blocked because they would destroy consistency of the internal data structures.

IStatusContainer manages a more complex data structure than just a BTree: it also provides for user and tag indexes. These are covered in additional methods.

is_most_recent_in_thread(statusupdate)

Returns True if this statusupdate is the most recent reply in its thread, or if it’s a toplevel post without replies.

Returns False if there are more recent statusupdates in this thread.

get()

Query for an attribute description

keys(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

context_values(context, min=None, max=None, limit=100)

Filter IStatusUpdate values by IMicroblogContext object. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

thread_keys(thread_id, min=None, max=None, limit=100)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns ids with the same toplevel thread parent (including parent)

context_items(context, min=None, max=None, limit=100)

Filter (key, IStatusUpdate) items by IMicroblogContext object. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items context <object> filters on StatusUpdates keyed to that context’s UUID.

iteritems(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

thread_values(thread_id, min=None, max=None, limit=100)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns updates with the same toplevel thread parent (including parent)

iterkeys(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

mention_keys(mentions, min=None, max=None, limit=100)

Filter IStatusUpdate keys by mentions. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

thread_items(thread_id, min=None, max=None, limit=100)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns items with the same toplevel thread parent (including parent)

context_keys(context, min=None, max=None, limit=100)

Filter IStatusUpdate keys by IMicroblogContext object. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

clear()

Empty the status storage and all indexes.

user_items(users, min=None, max=None, limit=100)

Filter (key, IStatusUpdate) items by iterable of userids. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

mention_values(mentions, min=None, max=None, limit=100)

Filter IStatusUpdate values by mentions. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

add(status)

Add a IStatusUpdate.

Actual storage may be queued for later insertion by the implementation.

Returns 1 on completion of synchronous insertion. Returns 0 when the actual insertion is queued for later processing.

values(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

user_values(users, min=None, max=None, limit=100)

Filter IStatusUpdate values by iterable of userids. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

itervalues(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

items(min=None, max=None, limit=100, tags=None, users=None)

BTree compatible accessor. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items returns matching for either users or tags, if given.

user_keys(users, min=None, max=None, limit=100)

Filter IStatusUpdate keys by iterable of userids. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

mention_items(mentions, min=None, max=None, limit=100)

Filter (key, IStatusUpdate) items by mentions. min and max are longint IStatusUpdate.id keys. limit returns [:limit] most recent items

interface ploneintranet.microblog.interfaces.IStatusUpdate

Bases: zope.interface.Interface

A single ‘tweet’.

attachments = <zope.interface.interface.Attribute object>

Returns an iterable IAttachmentStorage with the file attachment(s) for this IStatusUpdate. You can obtain the filenames by .attachments.keys() and get the file data by .attachments[key]

creator = <zope.schema._bootstrapfields.TextLine object>

Author name (for display)

text = <zope.schema._bootstrapfields.Text object>

add_statusupdate_button

tags = <zope.interface.interface.Attribute object>
userid = <zope.schema._bootstrapfields.TextLine object>

Userid

_content_context_uuid = <zope.interface.interface.Attribute object>

UUID of the object related to the status update

context_object = <zope.interface.interface.Attribute object>

UUID of context object (e.g. a Page)

thread_id = <zope.interface.interface.Attribute object>

status.id from parent

add_attachment(filename, data)

Add a binary attachment. Can be called multiple times to attach multiple files.

replies()

Return a list of replies (IStatusUpdate)

remove_attachment(filename)

Remove the attachment named <filename>

creation_date = <zope.schema._field.Date object>

Creation date

id = <zope.schema._bootstrapfields.Int object>

A longint unique status id

_microblog_context_uuid = <zope.interface.interface.Attribute object>

UUID of IMicroblogContext (e.g. a workspace)

absolute_url()

View this statusupdate in it’s proper context

interface ploneintranet.microblog.interfaces.IURLPreview

Bases: zope.interface.Interface

Tool to generate url preview image

generate_previews(url)

Return web preview image urls, in most cases it’s the OG link.

ploneintranet.microblog.migration module

ploneintranet.microblog.statuscontainer module

class ploneintranet.microblog.statuscontainer.BaseStatusContainer(context=None)

Bases: persistent.Persistent, ExplicitAcquirer

This implements IStatusUpdate storage, indexing and query logic.

This is just a base class, the actual IStorageContainer used in the implementation is the QueuedStatusContainer defined below.

StatusUpdates are stored in the private _status_mapping BTree. A subset of BTree accessors are exposed, see interfaces.py. StatusUpdates are keyed by longint microsecond ids.

Additionally, StatusUpdates are indexed by users and tags. These indexes use the same longint microsecond IStatusUpdate.id.

Special user_* prefixed accessors take an extra argument ‘users’, an interable of userids, and return IStatusUpdate keys, instances or items filtered by userids, in addition to the normal min/max statusid filters.

For backward compatibility sake, microblog_context is indexed as _uuid_mapping and accessors are called .context_*. This microblog_context is the security context for statusupdates.

The newer content_context is indexed as _content_mapping and accessors are called .content_*. This has no security impact and is only a convenience for per-document accessors.

_blacklist_microblogcontext_uuids()

Returns the uuids for all IMicroblogContext that the current user has no access to.

All the read accessors rely on this method for security checks.

_check_add_permission(status)
_check_delete_permission(status)

StatusUpdates have no local ‘owner’ role. Instead we check against permissions on the microblog context and compare with the creator.

_check_status(status)
_context2uuid(context)
_get(key)
_has_global_view_statusupdate_permission()
_idx_content_context(status)
_idx_is_content(status)
_idx_is_human(status)
_idx_mentions(status)
_idx_microblog_context(status)
_idx_tag(status)

Update the StatusContainer tag index with any new tags :param status: a StatusUpdate object

_idx_threadid(status)
_idx_user(status)
_notify_add(status)
_query_mapping(mapping, keys)

Calculate the union of all statusids indexed in <mapping> on any of the <keys>. Always returns an LLTreeSet ready for further processing.

_store(status)
_unidx(id)
_unidx_content_context(status)
_unidx_is_content(status)
_unidx_is_human(status)
_unidx_mentions(status)
_unidx_microblog_context(status)
_unidx_tag(status)
_unidx_threadid(status)
_unidx_user(status)
_update_ctime()

Update _ctime for cache busting. Requires *micro*seconds granularity to avoid RuntimeError by cache interference. Only used for cache invalidation.

_update_mtime()

Update _mtime on statusupdate add. Uses milliseconds. This flag is used for the cache key and for the async time window.

_whitelist_microblogcontext_uuids()

Current catalog query implicitly filters on View permission for the current user. We should not rely on View adequately representing ViewStatusUpdate.

The current implementation takes a conservative approach by applying an extra explicit security check for ViewStatusUpdate.

It is theoretically possible that the result excludes workspaces for which the user does have ViewStatusUpdate but does not have View.

A possible performance optimization that would also fix the overly conservative bias would be to add a special index for ViewStatusUpdate and use that directly in the catalog query. See http://stackoverflow.com/questions/23950860/how-to-list-users-having-review-permission-on-content-object

However, the number of IMicroblogContext objects in a site is normally quite limited and the outcome of this check is cached per request, which should hopefully limit the performance cost.

EDIT: this is costly, see workaround and comments below.

add(status)
allowed_status_keys(*args, **kwargs)

Return the subset of IStatusUpdate keys that are related to UUIDs of accessible contexts. I.e. blacklist all IStatusUpdate that has a context which we don’t have permission to access.

This is the key security protection used by all getters. Because it’s called a lot we’re caching results per user request. Because we use ram caching, we must make sure the return value is not a ZODB BTree accessor, so we cast to tuple.

allowed_status_keys_extranet()
allowed_status_keys_intranet()
clear()
content_items(content_context)
content_keys(content_context)
content_values(content_context)
context_items(microblog_context, min=None, max=None, limit=100, nested=True)
context_keys(microblog_context, min=None, max=None, limit=100, nested=True)
context_values(microblog_context, min=None, max=None, limit=100, nested=True)
delete(id, restricted=True)
dontcache = False
get(key)
insert(key, value)
is_content_items(min=None, max=None, limit=100)
is_content_keys(min=None, max=None, limit=100)
is_content_values(min=None, max=None, limit=100)
is_human_items(min=None, max=None, limit=100)
is_human_keys(min=None, max=None, limit=100)
is_human_values(min=None, max=None, limit=100)
is_most_recent_in_thread(statusupdate, tags=None)
items(min=None, max=None, limit=100, tags=None, users=None)
iteritems(min=None, max=None, limit=100, tags=None, users=None)
iterkeys(min=None, max=None, limit=100, tags=None, users=None)
itervalues(min=None, max=None, limit=100, tags=None, users=None)
keys(min=None, max=None, limit=100, tags=None, users=None)
mention_items(mentions, min=None, max=None, limit=100)
mention_keys(mentions, min=None, max=None, limit=100)
mention_values(mentions, min=None, max=None, limit=100)
nested_uuids(context)
pop(k, d=None)
secure(keyset)

Filter keyset to return only keys the current user may see.

NB this may return statusupdates with a microblog_context (workspace) accessible to the user, but referencing a content_context (document) which the user may not access yet because of content workflow.

Filtering that is quite costly and not done here - instead there’s a postprocessing filter in activitystream just before rendering.

setdefault(k, d)
thread_items(thread_id, min=None, max=None, limit=100)
thread_keys(thread_id, min=None, max=None, limit=100)
thread_values(thread_id, min=None, max=None, limit=100)
update(collection)
user_items(users, min=None, max=None, limit=100)
user_keys(users, min=None, max=None, limit=100)
user_values(users, min=None, max=None, limit=100)
values(min=None, max=None, limit=100, tags=None, users=None)
class ploneintranet.microblog.statuscontainer.QueuedStatusContainer(context=None)

Bases: ploneintranet.microblog.statuscontainer.BaseStatusContainer

A write performance optimized IStatusContainer.

This separates the queuing logic from the base class to make the code more readable (and testable).

For performance reasons, an in-memory STATUSQUEUE is used. StatusContainer.add() puts StatusUpdates into the queue.

MAX_QUEUE_AGE is the commit window in milliseconds. To disable batch queuing, set MAX_QUEUE_AGE = 0

.add() calls .autoflush(), which flushes the queue when ._mtime is longer than MAX_QUEUE_AGE ago.

So each .add() checks the queue. In a low-traffic site this will result in immediate disk writes (msg frequency < timeout). In a high-traffic site this will result on one write per timeout, which makes it possible to attain > 100 status update inserts per second.

Note that the algorithm is structured in such a way, that the system automatically adapts to low/high traffic conditions.

Additionally, a non-interactive queue flush is set up via _schedule_flush() which uses a volatile thread timer _v_timer to set up a non-interactive queue flush. This ensures that the “last Tweet of the day” also gets committed to disk.

An attempt is made to make self._mtime and self._v_timer thread-safe. These function as a kind of ad-hoc locking mechanism so that only one thread at a time is flushing the memory queue into persistent storage.

ASYNC = False
MAX_QUEUE_AGE = 1000
_autoflush()
_queue(status)
_schedule_flush()

A fallback queue flusher that runs without user interactions

_scheduled_autoflush(site_path=None, environ=None)

This method is run from the timer, outside a normal request scope. This requires an explicit commit on db write

add(status)
flush_queue()
ploneintranet.microblog.statuscontainer.cache_key(method, self)

Used as ramcache key for the expensive and frequently used allowed_status_keys() results. - cache per user - until a new update is inserted - for maximally 1 minute

The short time interval is needed in case the user’s workspace memberships change - this should invalidate the cache but we’re not listening to that event directly. One minute on the other hand is enough to cache the results for multiple calls during a single page rendering request - which should take seconds rather than a minute, but real life can be slow (especially if the cache expires, which has dramatic effects…)

memoize.ram automatically garbage collects the cache after 24 hours.

ploneintranet.microblog.statuscontainer.getZope2App(*args, **kwargs)

Gets the Zope2 app.

Copied almost verbatim from collective.celery

ploneintranet.microblog.statusupdate module

class ploneintranet.microblog.statusupdate.StatusUpdate(text=u'', microblog_context=None, thread_id=None, mention_ids=None, tags=None, content_context=None, action_verb=None)

Bases: persistent.Persistent

Title()
_context2uuid(context)
_init_content_context(thread_id, content_context)

We store the uuid as a reference of a content_context related to this status update

_init_creator()
_init_mentions(mention_ids)
_init_microblog_context(thread_id, microblog_context=None, content_context=None)

Set the right security context. If thread_id is given, the context of the thread parent is used and the given context arg is ignored.

E.g. a reply globally to a parent post done in a workspace takes the security context of the parent post.

_init_userid()
_uuid2context(uuid=None)
_uuid2object(uuid)
absolute_url()
action_verb

Backward compatible accessor

add_attachment(filename, data)

Add a binary attachment. Can be called multiple times to attach multiple files

Parameters:
  • filename – name of the file to attach
  • filename – file data to attach
attachments

The attachment storage. Lists filenames via .keys().

can_delete

StatusUpdates have no local ‘owner’ role. Instead we check against permissions on the microblog context and compare with the creator.

can_edit

StatusUpdates have no local ‘owner’ role. Instead we check against permissions on the microblog context and compare with the creator.

content_context
delete()
edit(text)

keeps original text across multiple edits

edited

Return last edit date if modified, or None if never changed.

getId()
getObject()
getURL()
is_content_update

Show content updates, including replies on toplevel content updates.

is_human_update

A ‘human’ update is either a toplevel post without content_context, or a reply (even a reply on a content update). Or, possibly, a stream post with attachment that got converted to a content update.

microblog_context
original_text

Return original text of a (multiply) edited update.

remove_attachment(filename)

Remove the attachment named <filename>

Parameters:filename – name of the file to remove
replies()

ploneintranet.microblog.testing module

ploneintranet.microblog.tool module

class ploneintranet.microblog.tool.MicroblogTool(context=None)

Bases: Products.CMFCore.utils.UniqueObject, OFS.SimpleItem.SimpleItem, ploneintranet.microblog.statuscontainer.QueuedStatusContainer

Provide IStatusContainer as a site utility.

id = 'ploneintranet_microblog'
meta_type = 'ploneintranet.microblog tool'

ploneintranet.microblog.urlpreview module

class ploneintranet.microblog.urlpreview.URLPreview(context)

Bases: object

generate_preview(url)

ploneintranet.microblog.utils module

ploneintranet.microblog.utils._longkeysortreverse_direct(accessor, minv, maxv, limit)

minv or limit is None: do not optimize

ploneintranet.microblog.utils._longkeysortreverse_optimized(accessor, maxv, limit)

not minv and limit is not None: Optimize by winding backward until limit is reached. This is the normal scenario: walking back 15 at a time.

ploneintranet.microblog.utils.longkeysortreverse(btreeish, minv=None, maxv=None, limit=None)

Performance optimized keyspace accessor. Returns an iterable of btreeish keys, reverse sorted by key. Expects a btreeish with long(microsec) keys.

In case a limit, but neither minv nor maxv is given, optimizates by not sorting the whole keyspace, but instead heuristically chunk the keyspace and sort only chunks, until the limit is reached.

The reason for this is that we want the most recent slice, which is last in the accessor, so we cannot just start iterating the slice. Basically we want to iterate backwards.

Module contents