Requirements / FAQ¶

Data Management Requirements¶

• How do you intend to use this media stream data now and in the future? What are your ultimate goals?

  Beyond your need to simply store video stream data, it helps our ingestion process to understand how the data is intended to be used, and in what context. What does the stream contain? Why is it important to ingest? What research is made possible by capturing this video stream data?

• How do others (your colleagues, your organization, other organizations) intend to consume this media stream data? And who are those other consumers?

  Outside your own immediate needs, are there others who will need access to your video stream data? And do they have the same goals as you? And if not, should those goals still be considered as a part of this ingestion process?

• Do you own the physical devices responsible for video streaming, or do you have permission from an entity to use this video stream? What is their contact information?

  It’s important to understand who owns or is ultimately responsible for cameras and the video content they provide.

• If different than the owners of the physical devices, what is the contact information for those responsible for the operation and maintenance of the devices?

  This WebCOOS project and its partners will not support camera devices directly, only consume the media streams emitted by the devices as configured by device owners / operators. As such, we will need to interface with individuals or groups responsible for the technical operation of the cameras in the event of technical issues or failure.

• What efforts have taken place, or are taking place, for the video stream data in question? Where have those efforts succeeded? Where have they not?

  It helps us to know if there are past or current projects that deal with your video stream data, especially if those projects may be in conflict with our ingestion, or our ingestion depends on the outcome of another project.

• Are there other efforts that WebCOOS is a part of related to this video stream data? Will this video ingestion need to relate or integrate with those efforts?

  If you have worked with us in the past, or are familiar with our infrastructure, it helps to know very early on if there are requirements for data ingestion tied to those existing efforts. A common integration would be to include your media stream content on your organization’s portal that WebCOOS or its partners already host.

• Will the ingested video stream need to be available to individuals? To organizations? Or does this data need to be freely available on the Internet?

  By default and by preference, all data in the WebCOOS data system if available publicly. It’s possible to make video stream data available to specific individuals or groups. However, additional effort, resources, and configurations will be required.

• What is the predicted lifespan of this video stream data?

  Is there a maximum lifespan for your data? And what level of access will be required of older, archived data?

• Do you need to provide real-time or near real-time video streaming?

  If you require public access to your media stream, particularly live or real-time access, it is recommended to use our stream ingestion and hosting infrastructure to support the stream. If required, a live version of your stream can be provided for adaptive streaming clients, allowing stream consumers to select higher or lower levels of stream quality based on their hardware limitations, network limitations, or preferences.

Device Requirements¶

Regardless of how WebCOOS receives your data there are questions that should be considered for each device, or class of devices, that you intend to ingest with our infrastructure.

• What media / picture formats does your device support?

  The WebCOOS webcam portal supports, at a minimum, the following picture formats:

  • H264 (AVC) video

  • H265 (HEVC) video

  Other formats may be supported, but analysis will need to be done on a case-by-case basis.

• What are the desired picture formats, frame sizes, picture quality, or time division requirements for the final video product?

  For the final media product (converted and rendered video files, or other media products), is there a level of quality that must be maintained? For example, you might need to capture the full 1080p video stream outputted by your camera, and have the stream captured with the same quality, but split into 10 minute video segments.

  Alternatively, is there a reduction in quality (smaller frame size, lower bit rate, slower frame rate) that is required? For example, you may only be interested in 1 frame per second, or 1 frame per minute to be captured from a stream that is transmitted at 24 frames per second from the device.

  Note that it is very possible to have more than one media product output, with all products simultaneously “muxed” with different parameters.

Streaming Requirements¶

For networked devices capable of producing media (video) streams, we will need the following information:

• What is the make and model of each device, or class of device?

  In the event that we must troubleshoot issues specific to a device’s configuration, or implementation of a protocol, it is immensely helpful to know the basic make and model of the device.

• How many video devices will need to be ingested?

  More devices mean larger storage, computation, and bandwidth resources. Knowing the number of devices or potential future devices will give us a sense of how best to plan our stream ingestion infrastructure.

• Is the device available via a streaming network protocol (such as RTSP?)

  If the device’s media stream is available via RTSP, then our current infrastructure is capable of ingesting its streaming video.

  If RTSP is not supported, we will have to evaluate the protocols available on the camera on a case-by-case basis.

• Are there any network restrictions like firewalls, NAT configurations, or bandwidth caps that will affect streaming over the device’s network?

  Media streaming tends to introduce unique problems when it comes to computer networks. Where websites are based on the transmission of short and well-defined documents and files that have a clear beginning and end, media streams are comprised of media fragments, continuously transmitted without end. Depending on the quality of the signal, media streams also introduce a heavy, continuous load on a network, which can affect other services and users on that network.

  The RTSP protocol, which we use to consume live media streams, uses both TCP and UDP. These are supported by any modern networking device, but aspects of the RTSP protocol often requires extra configuration. Specifically, we will need to connect directly to the device on both:

  • …a TCP port, for the purposes of controlling aspects of the media stream)_

  • …and a range of UDP ports, for the purposes of transmitting and receiving media stream data._

  If both the TCP and the UDP ports cannot be made accessible from a WebCOOS host, an analysis will need to be done to determine the best way to maintain network connections to your media device.

• What are the hostnames or IP addresses at which these devices can be reached? What TCP or UDP port numbers?

  These should be IP addresses or hostnames that publicly accessible via the Internet, or via a network configuration that is provided to us (a virtual private network, or VPN, for example).

• Are there any security requirements for connecting to the media device?

  Often, in addition to requiring the credentials necessary to administer a device, there are also credentials to access media streams to prevent unauthorized access or to limit the number of consumers that connect to a device.

  The RTSP protocol provides for username and password authentication, as well as supports secure transport over TLS (RTSPS). Credentials are recommended to protect your device, and secure transport to ensure those credentials are not compromised.

  Additionally, there may be firewall rules specific to your device that restrict network access only from certain sources (sometimes referred to as “source IP filters”). These rules will need to be modified or amended to support access from one or more of our IP addresses.

• Are there any device limitations, or resource restrictions that an ingestion process should be aware of?

  Most consumer-grade cameras are only designed to support a handful of simultaneous users. As such, we will want to ensure that our ingestion process respects these limitations.

  Other restriction examples:

  • Your device is only available during certain hours.

  • Your device’s hardware/software only supports a certain number of connections.

  • Your camera can only produce certain formats or certain stream qualities.

• What are the desired picture formats, frame sizes, picture quality, or time division requirements for the final video product?

  For the final media product (converted and rendered video files, or other media products), is there a level of quality that must be maintained? For example, you might need to capture the full 1080p video stream outputted by your camera, and have the stream captured with the same quality, but split into 10 minute video segments.

  Alternatively, is there a reduction in quality (smaller frame size, lower bit rate, slower frame rate) that is required? For example, you may only be interested in 1 frame per second, or 1 frame per minute to be captured from a stream that is transmitted at 24 frames per second from the device.

  Note that it is very possible to have more than one media product output, with all products simultaneously “muxed” with different parameters.

S3 Uploading Requirements¶

For non-streaming data uploads, we will need the following information:

• What is the volume of the archive data holdings? How much new data is produced each day?

  To resource appropriate hardware we will need to understand your data volume needs both historically and into the future. WebCOOS uses a distributed storage technology that allows us to scale to meet any volume need but there are cost implications to consider that we must take into account before uploading begins.

External S3 Indexing Requirements¶

For externally hosted datasets, we will need the following information in addition to the information above in the S3 Uploading Requirements section:

• Are you data organized in a manor so metadata can be extracted from file path or file names?

  When indexing data from an external provider, WebCOOS does it best not to download and analyze every individual file. We will try to extract as much information as possible from the file paths and names that your data is available through

  Some of the metadta fields we extract for streamed and uploaded data are as follows:

  • slug or unique name of the device (i.e. camera-01)

  • providing organization (i.e. noaa)

  • product name (i.e. video-clips, still-images, etc.)

  • duration/length of each video, extracted from the filename or ffprobe metadata

  • location of each video and image, extracted from the filename, ffprobe, or EXIF metadata

  • timestamp of each video and image, extracted from the filename, ffprobe, or EXIF metadata

• Is your data hosted through an S3 compatible API?

  If your data is not already in Amazon S3 or another S3 compatible API storage engine there are way to host a folder of files on any computer through an S3 compatible API using the Minio. WebCOOS may be able to help serve your data in a compaible manor so it can be index externally and available through the WebCOOS data system, please get in touch!