Bees

If you worked with XcooBee, you known that the XcooBee workflow is organized around bees. Bees can work alone or in hives to accomplish tasks. However, the paradigm is far more extensible.  Each Bee is a self-contained micro-service module. These service modules can be written by any developer willing to learn the module patterns.

Currently bee modules can be written in JavaScript running in a NodeJS environment. We support NodeJS 8.10. We are working towards making this available in C#, Java, and Python in later phases of XcooBee.

After your code is complete and tested you can submit your repository to us by contacting developers@ xcoobee.com. This process is manual at this point so please bear with us.

 

The Bee Module Life-cycle

Your code in essence will act as an independent node module (agent or plugin) which we at XcooBee refer to as bee. Bees run in a restricted environment with limitation on time and space (memory and disk). Bees can be run in sequence, in which case the sequence of executions is referred to as flightpath. The blueprint (class) of a multitude of Bees is referred to as Hive. Bees can communicate along the flightpath using the flightprocessing data object if needed. Each bee works independently on the input file provided to produce an output file for the next bee to process. The last processed output file is what is returned to the user or designated recipient.

The lifecylce consists of three steps

• invocation
• execution
• shutdown

invocation

The XcooBee system will invoke the flight() method of your primary code file for your module as indicated in your module’s package.json file. It will use the following signature:

flight(services, data, callback) {

}

During the invocation services and data arguments are populated. They allow you to read the input file and get insight into the environment.

execution

The execution step is left to your imagination. Anything you wish to do with the input file can be done given that you stay inside the restrictions for processing time, memory and disk-space. The current limits are

• time:
  • small instance: 25s
  • medium instance: 140s
  • large instance: 280s
• memory
  • small: 192 MB
  • medium: 1024 MB
  • large: 1536 MB
• disk: 512 MB

shutdown

An orderly shutdown occurs when you call the callback() function that was passed to you during invokation at the end of your processing. You can call it with either an error or success message. In case of error the XcooBee system will retry your bee two more times. If you do not complete your processing in the time allocated for your instance it is considered an unorderly shutdown. Your process will be terminated and XcooBee will handle it as an Error callback.

Success Callback Example

callback(null,"success message");

Error Callback Example

callback("problem occurred with file",null);

Services

Each bee when invoked will be passed a collection of services and a collection of data to conduct its operations. This is passed as services argument during the flight() function invocation.

log(message, type)

You can use the log to create trace entries for your processing. Use this service to write file processing related feedback. This is generally not displayed to the end-user but available to XcooBee support. where message = string: the message to be logged where type = enum one of (info|warning|error)

writeStream()

default write stream If your bee produces output as part of processing you can use this the default writeStream() to stream it to disk. If you need a specific filename, you can use the writeStreamManager() to create one as an alternative.

readStream()

default read stream This stream has access to the input file for your process.

writeStreamManager(filename, type)

type: wip | bee_output

factory to add write streams if bee has the option set. You can use this to create multiple files on the processing disk. You can create work in process files or output files based on the type selection.

timeToRun()

information on how much time is left before the bee will be shut down. The metric is in milliseconds.

addParam(key,value)

Add a parameter to the flightprocessing object. This is how we can communicate with downstream bees on the flight path. Add data to parameter structure to be provided to next bee (changes the directive file, see directive file in developer documentation).

getNextId()

Sequence generator. Will return next available integer. This starts with 1.

mail(template)

You can attempt to send email to user. To do so you will need to know the XcooBee mail template reference and replacement values. More details for this in the future. This also subject to users notification preferences.

The data object

When your bee is invoked a certain set of data is made available to it via the data argument in your function definition [flight(services, data, callback)]. This allows basic information to flow to the bee for processing and it has several subkeys. All subkeys are optional.

The subkeys are: integrations, user_data, parameters, flightprocessing, and env

integrations

If the bee requires access tokens from the XcooBee platform and the user has authorized it, the XcooBee platform will populate this node with specific information needed for each integration. The structure of object will depend on the integration accessed.

user_data

Basic data about the user such as name and XcooBeeId. This will also contain the user’s external reference passed in via userReference element of the bee-directive file.

parameters

The processing parameters that were provided by the user during the hiring process of the bee and written in to the bee-directive file.

flightprocessing

This is a communication object shared by all bees. Bees can add data using the addParam(key, value) service. Bees can read all data saved by previous bees (for multi-step processing). They can write new data. The can only override their own data (data that this bee has written).

env

This node contains basic environment information such as the path to work files and output files. This can be used by programs to directly place files into them. However, this is not recommend practice.

Module Construction

A minimal set of rules should be followed when you write a bee module:

• you should use eslint generally following the AirBnB rule set. We at XcooBee have a few changes that are made available in the .eslintrc example file in the example project.
• you should have test coverage for at least 80% of your code
• Provide an SVG graphic that will act as your bee visual identifier.
• Provide bee name and descriptive text in at least one of the supported languages for XcooBee.

 

Sample Bee

You can see a sample implementation of a bee in this public repo. This is not meant to be comprehensive or fully showing all elements of the bee module API. It should, however, be able to get you started.

https://github.com/XcooBee/test-bee-simple

Testing your Work

We have a test utility that you can include in your projects to run them locally and observe and fix issues in your development environment of choice.

https://github.com/XcooBee/bee-test-utility

 

Review Requirement

After your code is complete and tested you can submit your repository to us by contacting developers@ xcoobee.com. This process is manual at this point so please bear with us.

Bee Module for general use:

Great thanks for contributing to the community and making improvements. You will need to publish your bees under MIT or Apache license for us to use it. We will run it through analysis in terms of infrastructure cost and assess basic points appropriate for the use.

For private or corporate use:

If you are a business organization and are creating a bee for internal use only, there might be service cost assessment. We will be in touch to discuss all the remaining details such as scope of your bee (who can use it, at what cost, etc.). You will also have to agree to a few legal terms before we can publish the bee.