Skip to content

AI Application Porting Guide

This guide walks through porting a Linux AI application to Android, covering the main architectural changes and platform-specific considerations.

Contents

1. Architecture Planning

Before starting, analyze your Linux application to determine what should be ported to Java and what stays in C++.

Keep in C++ (via JNI): - CPU-intensive post-processing with heavy computation loops - Complex algorithms that are difficult to rewrite - Existing tested code with many dependencies - Code that manipulates large data structures

Port to Java: - Camera input and frame capture - UI rendering and user interaction - Application lifecycle and threading management - TVM Runtime - DRP-AI Driver calls (Use the Java driver HAL interface)

Android Constraints:

Android imposes several restrictions that differ from Linux environments. Thread stack size is limited to 1MB by default, so large buffers must be allocated on the heap rather than the stack. File system access is sandboxed—applications cannot directly read files from arbitrary paths like /home/user/. Instead, resources must be bundled in the APK's assets folder and copied to internal storage. The camera is accessed through the Android HAL, which may require format conversions that weren't necessary with V4L2.

2. Create the Android Project

In Android Studio, select "Native C++" when creating a new project if you'll be using JNI. This automatically configures the NDK build system and creates the necessary folder structure.

Your project should follow this structure:

app/src/main/
├── assets/              # Model files and resources
├── cpp/                 # Native C++ code
│   ├── CMakeLists.txt
│   └── *.cpp
└── java/                # Android application code

Configure the CMakeLists.txt file to build your native code through Gradle. Link against required Android libraries like log for logging support.

cmake_minimum_required(VERSION X.XX.X)
add_library(your_lib SHARED native_code.cpp)
target_link_libraries(your_lib android log)

Add the TVM Runtime library to your build by placing drpruntime.aar in the app/libs/ folder and updating build.gradle.kts:

dependencies {
    implementation(files("libs/drpruntime.aar"))
}

3. Camera Capture with Camera2

Replace V4L2 with Android's Camera2 API. The fundamental difference is that V4L2 provides direct hardware access with minimal abstraction, while Camera2 routes everything through the Android Camera HAL. This means format negotiation works differently, and you may not get the exact pixel format your preprocessing expects. Depending on the camera, it might be possible to access the camera through a more low-level API if necessary.

Start by obtaining a CameraManager to enumerate available cameras. USB cameras typically appear with the LENS_FACING_EXTERNAL characteristic. Once you've identified the correct camera, open it asynchronously using a StateCallback that runs on a thread.

CameraManager manager = (CameraManager) getSystemService(CAMERA_SERVICE);
manager.openCamera(cameraId, stateCallback, backgroundHandler);

Use ImageReader to capture frames from the camera. Unlike V4L2's direct buffer access, Camera2 delivers frames through callbacks. Configure the ImageReader with your desired resolution and format, then register a listener:

imageReader = ImageReader.newInstance(width, height, ImageFormat.YUV_420_888, 2);
imageReader.setOnImageAvailableListener(reader -> {
    Image image = reader.acquireLatestImage();
    // Process image
    image.close();  
}, backgroundHandler);

Format Conversion Requirements:

The Camera2 API typically outputs YUV_420_888 regardless of what the camera hardware natively supports. If your pipeline expects a different format (like YUYV), you must implement a conversion layer. This adds overhead but is unavoidable when using the standard Android camera stack.

The conversion involves reading the Image's planar YUV data and repacking it into the interleaved format your preprocessing requires. Extract the Y, U, and V planes from the Image object, then interleave them according to your target format's specification. This typically adds a few ms per frame.

4. TVM Runtime Integration

The TVM Runtime is provided as drpruntime.aar with Java wrapper classes. Unlike Linux where you directly link against .so files, Android requires binding to a HAL service that provides access to the hardware.

Service Binding:

The DRP-AI HAL exposes its functionality through Android's AIDL (Android Interface Definition Language) service. You must bind to this service before you can use PreRuntime or JniRuntime. This binding happens asynchronously, so initialize DRP-AI in the service connection callback.

Intent intent = new Intent().setComponent(new ComponentName(
    "vendor.renesas.hal.drpai",
    "vendor.renesas.hal.drpai.HalBindService"));
bindService(intent, connection, BIND_AUTO_CREATE);

private final ServiceConnection connection = new ServiceConnection() {
    public void onServiceConnected(ComponentName name, IBinder binder) {
        halService = IRenesasDrpai.Stub.asInterface(binder);
        // Now initialize DRP-AI on a background thread
        drpaiHandler.post(() -> initializeDrpai());
    }
};

Initialization Process:

Once the service is bound, create PreRuntime and JniRuntime instances. The PreRuntime constructor now requires the HAL service interface as a parameter. Load your preprocessing configuration from the model directory, then obtain the DRP-AI memory address and load your inference model.

preruntime = new PreRuntime(halService);
preruntime.Load(preprocessDir);
long drpaiMemAddr = preruntime.GetDrpaiStartAddress();

runtime = new JniRuntime();
runtime.LoadModel(modelDir, drpaiMemAddr);

Running Inference:

The inference flow is similar to Linux, but note that you must call GetOutputBuffer() after preprocessing to retrieve the result buffer. This buffer then becomes the input for the inference stage.

// Create preprocessing parameter with input data
PreRuntime.SPreprocParam param = preruntime.CreatePreprocParam(inputSize);
param.inputBuffer.put(inputData);

// Run preprocessing
preruntime.Pre(param, outBuffer, outSize);
ByteBuffer prepOutput = preruntime.GetOutputBuffer();  

// Run inference
FloatBuffer floatInput = prepOutput.asFloatBuffer();
runtime.SetInput(0, floatInput);
runtime.Run(frequencyParam);

// Retrieve outputs
JniRuntime.OutputData output = runtime.GetOutput(index);

All DRP-AI operations should run on a dedicated background thread, never on the UI thread. If processing runs on the UI thread it can cause the app to freeze.

5. Handle Model Files

Assets bundled in the APK are compressed and cannot be accessed as regular files. The native code and TVM Runtime require actual file paths, so you must copy assets to internal storage on first run.

Implement a recursive copy function that checks each asset path. Android provides getFilesDir() for app-private storage that persists across app launches.

File destDir = new File(getFilesDir(), "models");
if (!destDir.exists()) {
    AssetManager am = getAssets();
    // Copy recursively from assets to destDir
}

Store the absolute path to the copied model directory with something like modelDir = new File(getFilesDir(), "yolov8n").getAbsolutePath(); and pass it to the TVM (pre)runtime. Check if files already exist before copying to avoid unnecessary copies after the first app launch. The internal storage path will be something like /data/data/your.package/files/models/, which can be passed directly to native code or Runtime.

6. Post-Processing with JNI

For performance-critical post-processing code, use JNI to call existing C++ implementations rather than rewriting in Java. This is particularly beneficial for preprocessing or postprocessing operations, like NMS or coordinate transformation, that involve loops over large datasets.

Java Declaration:

Declare native methods in your Java class and load the shared library. Create wrapper methods that prepare data for the native call and handle results. Here is a simple example function.

public class ProcessingHelper {
    static {
        System.loadLibrary("your_native_lib");
    }
    
    private native float[] nativePostProcess(
        float[] modelOutput, 
        int outputSize,
        int imgWidth, 
        int imgHeight);
}

Native Implementation:

In your C++ code, implement the JNI function with the full mangled name that follows Java's package structure. Extract data from Java arrays using i.e GetFloatArrayElements, process it, and return results as Java objects.

extern "C" {
JNIEXPORT jfloatArray JNICALL
Java_com_package_ProcessingHelper_nativePostProcess(
        JNIEnv *env, jobject thiz,
        jfloatArray input, jint size, jint width, jint height) {
    
    jfloat *inputData = env->GetFloatArrayElements(input, nullptr);
    
    float *results = new float[resultSize];
    
    // Process data...
    
    // Release Java array
    env->ReleaseFloatArrayElements(input, inputData, JNI_ABORT);
    
    // Create and return Java array
    jfloatArray outputArray = env->NewFloatArray(resultSize);
    env->SetFloatArrayRegion(outputArray, 0, resultSize, results);
    
    delete[] results;
    return outputArray;
}
}

Memory Management Critical Points:

Android's thread stack limit of 1MB will cause crashes if you allocate large arrays on the stack, so allocate them on the heap. Always release Java elements using i.e. ReleaseFloatArrayElements to prevent memory leaks. The JNI_ABORT flag tells the JVM you haven't modified the array, avoiding an unnecessary copy back.

For objects that persist across multiple frames (like pre-allocated buffers), consider creating them once and passing a handle (cast to jlong) between Java and C++. This eliminates per-frame allocation overhead.

7. Visualization with OpenCV

OpenCV on Android uses Java bindings instead of the C++ API. The method names and class hierarchy are similar but adapted to Java conventions.

Initialization:

Load OpenCV at application startup. The OpenCVLoader.initLocal() method loads the native OpenCV library from your APK.

import org.opencv.android.OpenCVLoader;

if (!OpenCVLoader.initLocal()) {
    // Handle initialization failure
}

Drawing Operations:

Convert Android Bitmaps to OpenCV Mat objects using the Utils class. Drawing functions are in the Imgproc class rather than directly on Mat.

import org.opencv.android.Utils;
import org.opencv.imgproc.Imgproc;
import org.opencv.core.Scalar;

Mat mat = new Mat();
Utils.bitmapToMat(bitmap, mat);

// Draw rectangle
Imgproc.rectangle(mat, 
    new Point(x1, y1), 
    new Point(x2, y2),
    new Scalar(255, 0, 0), 
    thickness);

// Draw text
Imgproc.putText(mat, text, new Point(x, y),
    Imgproc.FONT_HERSHEY_SIMPLEX, scale, color, thickness);

// Convert back to Bitmap
Utils.matToBitmap(mat, resultBitmap);

For best performance, pre-allocate Mat objects and reuse them across frames rather than creating new ones each time. Bitmaps should be recycled when no longer needed to free up memory, especially when processing video frames continuously.

8. Threading and Lifecycle Management

Android requires explicit thread management and proper cleanup to prevent resource leaks and crashes. Android applications must respond to lifecycle events and perform operations on appropriate threads.

Background Thread Setup:

Create dedicated HandlerThread instances for different tasks. The camera callback, inference processing, and DRP-AI operations should each run on separate background threads. Only UI updates should happen on the main thread.

HandlerThread cameraThread = new HandlerThread("Camera");
cameraThread.start();
Handler cameraHandler = new Handler(cameraThread.getLooper());

HandlerThread inferenceThread = new HandlerThread("Inference");
inferenceThread.start();
Handler inferenceHandler = new Handler(inferenceThread.getLooper());

Post work to these handlers instead of running it directly. This prevents blocking the UI thread and allows Android to manage thread priorities appropriately.

inferenceHandler.post(() -> {
    // Run inference
    Results results = processFrame(input);
    
    // Update UI on main thread
    runOnUiThread(() -> displayResults(results));
});

Resource Cleanup:

Implement proper cleanup in onDestroy() to release all resources when the application exits. This includes closing camera sessions, stopping background threads, releasing DRP-AI resources, and unbinding from services.

Close camera-related objects first, then stop threads by calling quitSafely() and waiting for them to finish with join(). Release PreRuntime and any native objects you created. Finally, unbind from the DRP-AI HAL service.

@Override
protected void onDestroy() {
    // Close camera
    if (imageReader != null) imageReader.close();
    if (cameraDevice != null) cameraDevice.close();
    
    // Stop threads
    if (inferenceThread != null) {
        inferenceThread.quitSafely();
        try { inferenceThread.join(); } catch (InterruptedException e) {}
    }
    
    // Release preruntime
    if (preruntime != null) preruntime.close();
    
    // Release native resources
    if (nativeHandle != 0) nativeCleanup(nativeHandle);
    
    // Unbind service
    unbindService(halServiceConnection);
    
    super.onDestroy();
}

Additional Considerations

DirectByteBuffer for Performance:

When passing large amounts of data between Java and native code, consider using DirectByteBuffer. Unlike regular arrays, DirectByteBuffers can be accessed from both Java and native code without copying, significantly reducing overhead for large data transfers.

Testing:

Use Android's Logcat for debugging. Test on the target hardware since emulators don't have access to the DRP-AI hardware or specialized camera formats.

For reference implementations, consult the AI applications.