Constructing pleasant Android digicam and media experiences

Posted by Donovan McMurray, Mayuri Khinvasara Khabya, Mozart Louis, and Nevin Mital – Developer Relations Engineers

Whats up Android Builders!

We’re the Android Developer Relations Digital camera & Media staff, and we’re excited to carry you one thing just a little totally different right this moment. Over the previous a number of months, we’ve been laborious at work writing pattern code and constructing demos that showcase reap the benefits of all the good potential Android presents for constructing pleasant consumer experiences.

A few of these efforts can be found so that you can discover now, and a few you’ll see later all year long, however for this weblog submit we thought we’d share among the learnings we gathered whereas going by this train.

Seize your favourite Android plush or rubber duck, and skim on to see what we’ve been as much as!

Future-proof your app with Jetpack

Nevin Mital

One among our focuses for the previous a number of years has been enhancing the developer instruments obtainable for video modifying on Android. This led to the creation of the Jetpack Media3 Transformer APIs, which supply options for each single-asset and multi-asset video modifying preview and export. In the present day, I’d prefer to concentrate on the Composition demo app, a pattern app that showcases among the multi-asset modifying experiences that Transformer allows.

I began by including a customized video compositor to exhibit how one can prepare enter video sequences into totally different layouts on your closing composition, akin to a 2×2 grid or a picture-in-picture overlay. You may customise this by implementing a VideoCompositorSettings and overriding the getOverlaySettings methodology. This object can then be set when constructing your Composition with setVideoCompositorSettings.

Right here is an instance for the 2×2 grid format:

object : VideoCompositorSettings {
  ...

  override enjoyable getOverlaySettings(inputId: Int, presentationTimeUs: Lengthy): OverlaySettings {
    return when (inputId) {
      0 -> { // First sequence is positioned within the high left
        StaticOverlaySettings.Builder()
          .setScale(0.5f, 0.5f)
          .setOverlayFrameAnchor(0f, 0f) // Center of overlay
          .setBackgroundFrameAnchor(-0.5f, 0.5f) // Prime-left part of background
          .construct()
      }

      1 -> { // Second sequence is positioned within the high proper
        StaticOverlaySettings.Builder()
          .setScale(0.5f, 0.5f)
          .setOverlayFrameAnchor(0f, 0f) // Center of overlay
          .setBackgroundFrameAnchor(0.5f, 0.5f) // Prime-right part of background
          .construct()
      }

      2 -> { // Third sequence is positioned within the backside left
        StaticOverlaySettings.Builder()
          .setScale(0.5f, 0.5f)
          .setOverlayFrameAnchor(0f, 0f) // Center of overlay
          .setBackgroundFrameAnchor(-0.5f, -0.5f) // Backside-left part of background
          .construct()
      }

      3 -> { // Fourth sequence is positioned within the backside proper
        StaticOverlaySettings.Builder()
          .setScale(0.5f, 0.5f)
          .setOverlayFrameAnchor(0f, 0f) // Center of overlay
          .setBackgroundFrameAnchor(0.5f, -0.5f) // Backside-right part of background
          .construct()
      }

      else -> {
        StaticOverlaySettings.Builder().construct()
      }
    }
  }
}

Since getOverlaySettings additionally offers a presentation time, we will even animate the format, akin to on this picture-in-picture instance:

Subsequent, I spent a while migrating the Composition demo app to make use of Jetpack Compose. With sophisticated modifying flows, it may possibly assist to reap the benefits of as a lot display screen house as is offered, so I made a decision to make use of the supporting pane adaptive format. This fashion, the consumer can fine-tune their video creation on the preview display screen, and export choices are solely proven on the similar time on a bigger show. Beneath, you’ll be able to see how the UI dynamically adapts to the display screen dimension on a foldable gadget, when switching from the outer display screen to the interior display screen and vice versa.

What’s nice is that through the use of Jetpack Media3 and Jetpack Compose, these options additionally carry over seamlessly to different gadgets and kind elements, akin to the brand new Android XR platform. Proper out-of-the-box, I used to be in a position to run the demo app in Residence House with the 2D UI I already had. And with some small updates, I used to be even in a position to adapt the UI particularly for XR with options akin to a number of panels, and to take additional benefit of the additional house, an Orbiter with playback controls for the modifying preview.

What’s nice is that through the use of Jetpack Media3 and Jetpack Compose, these options additionally carry over seamlessly to different gadgets and kind elements, akin to the brand new Android XR platform. Proper out-of-the-box, I used to be in a position to run the demo app in Residence House with the 2D UI I already had. And with some small updates, I used to be even in a position to adapt the UI particularly for XR with options akin to a number of panels, and to take additional benefit of the additional house, an Orbiter with playback controls for the modifying preview.

Orbiter(
  place = OrbiterEdge.Backside,
  offset = EdgeOffset.interior(offset = MaterialTheme.spacing.customary),
  alignment = Alignment.CenterHorizontally,
  form = SpatialRoundedCornerShape(CornerSize(28.dp))
) {
  Row (horizontalArrangement = Association.spacedBy(MaterialTheme.spacing.mini)) {
    // Playback management for rewinding by 10 seconds
    FilledTonalIconButton({ viewModel.seekBack(10_000L) }) {
      Icon(
        painter = painterResource(id = R.drawable.rewind_10),
        contentDescription = "Rewind by 10 seconds"
      )
    }
    // Playback management for play/pause
    FilledTonalIconButton({ viewModel.togglePlay() }) {
      Icon(
        painter = painterResource(id = R.drawable.rounded_play_pause_24),
        contentDescription = 
            if(viewModel.compositionPlayer.isPlaying) {
                "Pause preview playback"
            } else {
                "Resume preview playback"
            }
      )
    }
    // Playback management for forwarding by 10 seconds
    FilledTonalIconButton({ viewModel.seekForward(10_000L) }) {
      Icon(
        painter = painterResource(id = R.drawable.forward_10),
        contentDescription = "Ahead by 10 seconds"
      )
    }
  }
}

Jetpack libraries unlock premium performance incrementally

Donovan McMurray

Not solely do our Jetpack libraries have you ever coated by working constantly throughout current and future gadgets, however in addition they open the doorways to superior performance and customized behaviors to assist all sorts of app experiences. In a nutshell, our Jetpack libraries intention to make the frequent case very accessible and simple, and it has hooks for including extra customized options later.

We’ve labored with many apps who’ve switched to a Jetpack library, constructed the fundamentals, added their crucial customized options, and really saved developer time over their estimates. Let’s check out CameraX and the way this incremental improvement can supercharge your course of.

// Arrange CameraX app with preview and picture seize.
// Be aware: setting the decision selector is elective, and if not set,
// then a default 4:3 ratio will likely be used.
val aspectRatioStrategy = AspectRatioStrategy(
  AspectRatio.RATIO_16_9, AspectRatioStrategy.FALLBACK_RULE_NONE)
var resolutionSelector = ResolutionSelector.Builder()
  .setAspectRatioStrategy(aspectRatioStrategy)
  .construct()

non-public val previewUseCase = Preview.Builder()
  .setResolutionSelector(resolutionSelector)
  .construct()
non-public val imageCaptureUseCase = ImageCapture.Builder()
  .setResolutionSelector(resolutionSelector)
  .setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
  .construct()

val useCaseGroupBuilder = UseCaseGroup.Builder()
  .addUseCase(previewUseCase)
  .addUseCase(imageCaptureUseCase)

cameraProvider.unbindAll()

digicam = cameraProvider.bindToLifecycle(
  this,  // lifecycleOwner
  CameraSelector.DEFAULT_BACK_CAMERA,
  useCaseGroupBuilder.construct(),
)

After organising the fundamental construction for CameraX, you’ll be able to arrange a easy UI with a digicam preview and a shutter button. You should use the CameraX Viewfinder composable which shows a Preview stream from a CameraX SurfaceRequest.

// Create preview
Field(
  Modifier
    .background(Colour.Black)
    .fillMaxSize(),
  contentAlignment = Alignment.Heart,
) {
  surfaceRequest?.let {
    CameraXViewfinder(
      modifier = Modifier.fillMaxSize(),
      implementationMode = ImplementationMode.EXTERNAL,
      surfaceRequest = surfaceRequest,
     )
  }
  Button(
    onClick = onPhotoCapture,
    form = CircleShape,
    colours = ButtonDefaults.buttonColors(containerColor = Colour.White),
    modifier = Modifier
      .top(75.dp)
      .width(75.dp),
  )
}

enjoyable onPhotoCapture() {
  // Not proven: defining the ImageCapture.OutputFileOptions for
  // your saved photos
  imageCaptureUseCase.takePicture(
    outputOptions,
    ContextCompat.getMainExecutor(context),
    object : ImageCapture.OnImageSavedCallback {
      override enjoyable onError(exc: ImageCaptureException) {
        val msg = "Photograph seize failed."
        Toast.makeText(context, msg, Toast.LENGTH_SHORT).present()
      }

      override enjoyable onImageSaved(output: ImageCapture.OutputFileResults) {
        val savedUri = output.savedUri
        if (savedUri != null) {
          // Do one thing with the savedUri if wanted
        } else {
          val msg = "Photograph seize failed."
          Toast.makeText(context, msg, Toast.LENGTH_SHORT).present()
        }
      }
    },
  )
}

You’re already on observe for a strong digicam expertise, however what in case you needed so as to add some additional options on your customers? Including filters and results are straightforward with CameraX’s Media3 impact integration, which is one of many new options launched in CameraX 1.4.0.

Right here’s how easy it’s so as to add a black and white filter from Media3’s built-in results.

val media3Effect = Media3Effect(
  utility,
  PREVIEW or IMAGE_CAPTURE,
  ContextCompat.getMainExecutor(utility),
  {},
)
media3Effect.setEffects(listOf(RgbFilter.createGrayscaleFilter()))
useCaseGroupBuilder.addEffect(media3Effect)

The Media3Effect object takes a Context, a bitwise illustration of the use case constants for focused UseCases, an Executor, and an error listener. You then set the checklist of results you wish to apply. Lastly, you add the impact to the useCaseGroupBuilder we outlined earlier.

There are various different built-in results you’ll be able to add, too! See the Media3 Impact documentation for extra choices, like brightness, colour lookup tables (LUTs), distinction, blur, and lots of different results.

To take your results to yet one more degree, it’s additionally attainable to outline your personal results by implementing the GlEffect interface, which acts as a manufacturing facility of GlShaderPrograms. You may implement a BaseGlShaderProgram’s drawFrame() methodology to implement a customized impact of your personal. A minimal implementation ought to inform your graphics library to make use of its shader program, bind the shader program’s vertex attributes and uniforms, and situation a drawing command.

Jetpack libraries meet you the place you might be and your app’s wants. Whether or not that be a easy, fast-to-implement, and dependable implementation, or customized performance that helps the crucial consumer journeys in your app stand out from the remaining, Jetpack has you coated!

Jetpack presents a basis for revolutionary AI Options

Mayuri Khinvasara Khabya

Simply as Donovan demonstrated with CameraX for seize, Jetpack Media3 offers a dependable, customizable, and feature-rich answer for playback with ExoPlayer. The AI Samples app builds on this basis to thrill customers with useful and enriching AI-driven additions.

In right this moment’s quickly evolving digital panorama, customers anticipate extra from their media purposes. Merely enjoying movies is now not sufficient. Builders are consistently searching for methods to boost consumer experiences and supply deeper engagement. Leveraging the ability of Synthetic Intelligence (AI), notably when constructed upon sturdy media frameworks like Media3, presents thrilling alternatives. Let’s check out among the methods we will rework the way in which customers work together with video content material:

• Empowering Video Understanding: The core concept is to make use of AI, particularly multimodal fashions just like the Gemini Flash and Professional fashions, to investigate video content material and extract significant data. This goes past merely enjoying a video; it is about understanding what’s within the video and making that data readily accessible to the consumer.
• Actionable Insights: The purpose is to rework uncooked video into summaries, insights, and interactive experiences. This permits customers to rapidly grasp the content material of a video and discover particular data they want or be taught one thing new!
• Accessibility and Engagement: AI helps make movies extra accessible by offering options like summaries, translations, and descriptions. It additionally goals to extend consumer engagement by interactive options.

A Glimpse into AI-Powered Video Journeys

The next instance demonstrates potential video journies enhanced by synthetic intelligence. This pattern integrates a number of parts, akin to ExoPlayer and Transformer from Media3; the Firebase SDK (leveraging Vertex AI on Android); and Jetpack Compose, ViewModel, and StateFlow. The code will likely be obtainable quickly on Github.

There are two experiences specifically that I’d like to focus on:

• HDR Thumbnails: AI might help determine key moments within the video that would make for good thumbnails. With these timestamps, you need to use the brand new ExperimentalFrameExtractor API from Media3 to extract HDR thumbnails from movies, offering richer visible previews.
• Textual content-to-Speech: AI can be utilized to transform textual data derived from the video into spoken audio, enhancing accessibility. On Android you can even select to play audio in numerous languages and dialects thus enhancing personalization for a wider viewers.

Utilizing the appropriate AI answer

At the moment, solely cloud fashions assist video inputs, so we went forward with a cloud-based answer.Iintegrating Firebase in our pattern empowers the app to:

• Generate real-time, concise video summaries robotically.
• Produce complete content material metadata, together with chapter markers and related hashtags.
• Facilitate seamless multilingual content material translation.

So how do you really work together with a video and work with Gemini to course of it? First, ship your video as an enter parameter to your immediate:

val promptData =
"Summarize this video within the type of high 3-4 takeaways solely. Write within the type of bullet factors. Do not assume if you do not know"

val generativeModel = Firebase.vertexAI.generativeModel("gemini-2.0-flash")
_outputText.worth = OutputTextState.Loading

viewModelScope.launch(Dispatchers.IO) {
    attempt {
        val requestContent = content material {
            fileData(videoSource.toString(), "video/mp4")
            textual content(immediate)
        }
        val outputStringBuilder = StringBuilder()

        generativeModel.generateContentStream(requestContent).gather { response ->
            outputStringBuilder.append(response.textual content)
            _outputText.worth = OutputTextState.Success(outputStringBuilder.toString())
        }

        _outputText.worth = OutputTextState.Success(outputStringBuilder.toString())

    } catch (error: Exception) {
        _outputText.worth = error.localizedMessage?.let { OutputTextState.Error(it) }
    }
}

Discover there are two key parts right here:

• FileData: This element integrates a video into the question.
• Immediate: This asks the consumer what particular help they want from AI in relation to the supplied video.

After all, you’ll be able to finetune your immediate as per your necessities and get the responses accordingly.

In conclusion, by harnessing the capabilities of Jetpack Media3 and integrating AI options like Gemini by Firebase, you’ll be able to considerably elevate video experiences on Android. This mix allows superior options like video summaries, enriched metadata, and seamless multilingual translations, in the end enhancing accessibility and engagement for customers. As these applied sciences proceed to evolve, the potential for creating much more dynamic and clever video purposes is huge.

Go above-and-beyond with specialised APIs

Mozart Louis

Android 16 introduces the brand new audio PCM Offload mode which might cut back the ability consumption of audio playback in your app, resulting in longer playback time and elevated consumer engagement. Eliminating the ability nervousness drastically enhances the consumer expertise.

Oboe is Android’s premiere audio api that builders are in a position to make use of to create excessive efficiency, low latency audio apps. A brand new characteristic is being added to the Android NDK and Android 16 known as Native PCM Offload playback.

Offload playback helps save battery life when enjoying audio. It really works by sending a big chunk of audio to a particular a part of the gadget’s {hardware} (a DSP). This permits the CPU of the gadget to enter a low-power state whereas the DSP handles enjoying the sound. This works with uncompressed audio (like PCM) and compressed audio (like MP3 or AAC), the place the DSP additionally takes care of decoding.

This may end up in vital energy saving whereas enjoying again audio and is ideal for purposes that play audio within the background or whereas the display screen is off (suppose audiobooks, podcasts, music and so forth).

We created the pattern app PowerPlay to exhibit implement these options utilizing the most recent NDK model, C++ and Jetpack Compose.

Listed below are a very powerful elements!

First order of enterprise is to guarantee the gadget helps audio offload of the file attributes you want. Within the instance beneath, we’re checking if the gadget assist audio offload of stereo, float PCM file with a pattern charge of 48000Hz.

       val format = AudioFormat.Builder()
            .setEncoding(AudioFormat.ENCODING_PCM_FLOAT)
            .setSampleRate(48000)
            .setChannelMask(AudioFormat.CHANNEL_OUT_STEREO)
            .construct()

        val attributes =
            AudioAttributes.Builder()
                .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
                .setUsage(AudioAttributes.USAGE_MEDIA)
                .construct()
       
        val isOffloadSupported = 
            if (Construct.VERSION.SDK_INT >= Construct.VERSION_CODES.Q) {
                AudioManager.isOffloadedPlaybackSupported(format, attributes)
            } else {
                false
            }

        if (isOffloadSupported) {
            participant.initializeAudio(PerformanceMode::POWER_SAVING_OFFLOADED)
        }

As soon as we all know the gadget helps audio offload, we will confidently set the Oboe audio streams’ efficiency mode to the brand new efficiency mode choice, PerformanceMode::POWER_SAVING_OFFLOADED.

// Create an audio stream
        AudioStreamBuilder builder;
        builder.setChannelCount(mChannelCount);
        builder.setDataCallback(mDataCallback);
        builder.setFormat(AudioFormat::Float);
        builder.setSampleRate(48000);

        builder.setErrorCallback(mErrorCallback);
        builder.setPresentationCallback(mPresentationCallback);
        builder.setPerformanceMode(PerformanceMode::POWER_SAVING_OFFLOADED);
        builder.setFramesPerDataCallback(128);
        builder.setSharingMode(SharingMode::Unique);
           builder.setSampleRateConversionQuality(SampleRateConversionQuality::Medium);
        Consequence end result = builder.openStream(mAudioStream);

Now when audio is performed again, it will likely be offloading audio to the DSP, serving to save energy when enjoying again audio.

There’s extra to this characteristic that will likely be coated in a future weblog submit, totally detailing out the entire new obtainable APIs that can enable you to optimize your audio playback expertise!

What’s subsequent

After all, we had been solely in a position to share the tip of the iceberg with you right here, so to dive deeper into the samples, take a look at the next hyperlinks:

Hopefully these examples have impressed you to discover what new and interesting experiences you’ll be able to construct on Android. Tune in to our session at Google I/O in a pair weeks to be taught much more about use-cases supported by options like Jetpack CameraX and Jetpack Media3!