Sharing Media in Progressive Web Apps


Software Engineer
Blog, Frontend | 30th November 2018

Sharing Media in Progressive Web Apps

What are Progressive Web Apps (PWAs)?

With the widespread adoption of HTML5 standards, the lines between native and web applications are becoming more blurry. Now, increasing native features have become available for the browsers, making it difficult to decide whether to build a native or web application.

Here at NearForm we love pushing the boundaries of disruptive technology, and in a sense, this is what Progressive Web Apps (PWAs) are.

For more info see our other articles on PWAs: 

Building Progressive Web Apps

Payment Request API for online purchase in PWAs Using

Web Notifications to increase engagement in PWAs

Let’s remind ourselves what Progressive Web Apps are:

  • Progressive: they make use of the functionality that is available on your device. If the functionality is not available they provide a sensible fallback
  • Web: they are built with the web in mind so they are available for your device if it is web-ready.They run in a browser and the code base is essentially a web page
  • App: and last but not least, it is an App because it allows you to add it as a standalone bookmark to your website. It appears and opens as an App would but, unlike native Apps, the step of going to the App store and downloading the App is not required making it a more seamless and frictionless user experience

Building a Progressive Web App for Image Sharing

Given that the APIs available for web applications are very close to what the native version has to offer we decided to build a PWA that can take pictures and then share them via the native device methods.

DemoSource

In order to take the actual picture, we made use of the HTML5 `<video />` tag element. This allows us to use one of the devices `videoinput` sensors as a streaming source essentially piping all the input that is coming through via that stream to the `<video />` element in the UI. In order to access the video sensors on your device, you need to use the `navigator.mediaDevices` web API which has good support across browsers.

Sharing Media in Progressive Web Apps

What we need to do is select the desired `videoinput` sensor from the available mediaDevices and pipe that into the `<video />` HTML5 element.

For example, to list all available devices on an Android device you would print out:

`navigator.mediaDevices.enumerateDevices().then(devices => console.log(devices))`

And you would get a list similar to this:

Sharing Media in Progressive Web Apps

The list contains all the devices available for streaming input to your website, could be audio or video, but for taking pictures the `videoinput` kind is the one we want. We can see that the two available `videoinput` sensors are for the front and back camera, as they are labelled accordingly. In order to grab a picture from the stream what we want is essentially a freeze frame of the stream. To do that we want to draw to the canvas a frame from the video stream, then stop that stream.

`let context = canvasElement.getContext('2d')

  context.drawImage(videoPlayer, 0, 0, canvas.width, videoPlayer.videoHeight / (videoPlayer.videoWidth / canvas.width))

  videoPlayer.srcObject.getVideoTracks().forEach(track => {

    track.stop()

  })`

This gives us the image we want, but it is not in a usable format yet. If we want to upload it to a static asset hosting service then we need to convert it to a binary object, and with JS that would be a `Blob` type. After conversion we upload the file and retrieve the link, so now our next goal is to share it using the native share context menu.

Sharing Media in Progressive Web Apps

The Web Share API is not available on most browsers, so as we want to be Progressive about it we will use it where available.

Sharing Media in Progressive Web Apps

The problem now is that part of the devices that want to use the App will come from iOS so we need a sensible fallback for the lack of support.

A good workaround for this missing feature for iOS is using deep-links for sending data directly to the native apps and bypassing the context menu. On both iOS and Android, you can deeply link to an app by using a specific url, and you can send the app some url encoded data:

Eg:

`<a href="fb-messenger://share/?link=<url_encoded_content>;app_id=<app_id>"></a>`

You don’t have access to the native sharing context on iOS but you can build your own web components to share the link to your picture to the various native applications. This makes it more customizable but is different from the native share.

QR Code Scanning

The Media Streaming API gives us access to the video stream so we can do processing on the stream frames. We can use libraries to extract data from the frames, and there are various third-party libraries available for decoding different interesting information from images.

For demo purposes, we used QR codes as they provide a standard for encoding text in images.

We used a JS port of a popular library reading and generating QR codes. ZXing has ports into many languages and is a de facto standard for working with QR codes. The port used in our project is a smaller version that allows for detecting and decoding the QR from an image. The library used is jsQR.

What the library does is, it scans a given image and detects the position in the frame of a QR code if any, and if one is present it decodes it, returning the text data.

`new window.jsQR(currentImage.data, currentImage.width, currentImage.height)`

The approach we use for taking the pictures and also scanning the QR code allows us to go beyond only decoding, but we could also allow highlighting the QR code if one is detected, eg:

Sharing Media in Progressive Web Apps

Conclusion

We can use the Streaming Media API in real time and extract a lot of information from the video/pictures. We can decode standardized images and extract information. There are quite a few libraries that have been ported to JS that we can use for image processing and the HTML5 canvas is a very powerful tool that allows us to enhance them. QR code scanning is only a small example of the potential usage in PWAs.

The Web Share API is not yet mature enough to support the native sharing methods, but the fallback for the browser that doesn’t yet support it is acceptable for most use cases. It is only when the share context menu is highly custom that this is a realistic limitation. Sharing only works with text content so the sharing of binary content(images) is not supported via either the Web Share API or the deep-linking alternative.

That being said a deep-linking mechanism is a valid option and it works for both iOS and Android, so it can be used in older devices too. Most big companies have built PWAs to complement their existing native applications; Instagram, Facebook, Pinterest and Twitter just to mention a few. Even though the support for some of the features is not yet fully there we should use the feature if it’s there and provide a sensible fallback if it is not.

The code for the above app can be found here. For any further queries about PWA’s or whether you need to design and build new apps or modernise existing applications, please contact NearForm at info@nearform.com.

Top