Repository navigation
DPL Analysis: send CCDB tables together with the objects they refer to - #16079
Merged
Merged
Conversation
On a cache miss the analysis CCDB fetcher adopted the new object with CacheStrategy::Always, which caches and sends it, and then sent it a second time from AnalysisCCDBHelpers via adoptFromCache, which already sends every object referenced by the table once per timeframe. Add CacheStrategy::CacheOnly, which caches the message without sending it, and use it for the analysis fetcher, so that adoptFromCache is the only place its objects are sent from. The non-analysis CCDB fetcher keeps CacheStrategy::Always.
The analysis CCDB fetcher sent each CCDB table through the arrow backend and the CCDB objects it refers to, which keep them alive in the consumer, through the message backend. The two backends send separately, the arrow one first, and the consumer's DataRelayer completes the input on its first part. When every other input of the consumer was already there, as for the first timeframe, the slot was processed with the table alone and the objects then opened a new slot that could never complete, which was dropped with "Dropping incomplete AOD/GLOCCDBOBJ/1 ...". Serialise the table in the fetcher and send it through the message backend as well, followed by the objects. They then leave in one multipart, which the consumer relays as a whole before checking for completion, so they always land in the same slot with the table as the first part. The table is adopted with adoptContainer, like the objects, since messages created via snapshot are scheduled and would be sent after them. Its header and the arrow byte counters are set as the arrow backend does, so the rate limiting accounts for it as before. The alternatives were ruled out: - A separate input for the objects changes the topology of every analysis workflow using CCDB tables. - Not sending the objects at all and relying on the fetcher's cache would free an object still in use downstream when it is replaced, or when the fetcher exits before its consumers. - Swapping the order of the backends would make an object, when it arrives first, complete the input and be read as the table. Add a DataRelayer test for parts relayed from one multipart, which complete together with the table first, against parts relayed with a completion check in between, which reproduce the drop.
Member
Author
|
Currently broken. Should affect only the new Analysis CCDB fetcher. Merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The analysis CCDB fetcher sent each CCDB table through the arrow backend
and the CCDB objects it refers to, which keep them alive in the consumer,
through the message backend. The two backends send separately, the arrow
one first, and the consumer's DataRelayer completes the input on its
first part. When every other input of the consumer was already there, as
for the first timeframe, the slot was processed with the table alone and
the objects then opened a new slot that could never complete, which was
dropped with "Dropping incomplete AOD/GLOCCDBOBJ/1 ...".
Serialise the table in the fetcher and send it through the message
backend as well, followed by the objects. They then leave in one
multipart, which the consumer relays as a whole before checking for
completion, so they always land in the same slot with the table as the
first part. The table is adopted with adoptContainer, like the objects,
since messages created via snapshot are scheduled and would be sent after
them. Its header and the arrow byte counters are set as the arrow backend
does, so the rate limiting accounts for it as before.
The alternatives were ruled out:
analysis workflow using CCDB tables.
free an object still in use downstream when it is replaced, or when the
fetcher exits before its consumers.
arrives first, complete the input and be read as the table.
Add a DataRelayer test for parts relayed from one multipart, which
complete together with the table first, against parts relayed with a
completion check in between, which reproduce the drop.