{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":["admonition"]},"redocly_category":"Products","product_name":"Machine Learning","type":"markdown"},"seo":{"title":"Incremental Mini Batch Prediction","description":"Treasure Data Product Documentation · Collect and Unify · Segment and Activate · Experiment and Analyze · Decisioning Automate with AI Scale and Trust.","siteUrl":"https://docs.treasuredata.com","lang":"en-US","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"incremental-mini-batch-prediction","__idx":0},"children":["Incremental Mini Batch Prediction"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This document describes how to implement ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["incremental prediction"]}," and provides a best practice to cope with low latency requirements."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["There are three main patterns for prediction:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Online"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Offline"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Incremental"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"img","attributes":{"src":"/assets/247133042-d733e9c4-9139-4363-a652-65b3a7c5f714.a9bffd1c6133669aaf87179e1aee73b77f2ce418d0eec745d7b71606ea7d2e92.3cb60505.png","alt":""},"children":[]}]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Online prediction:"]}," Typically implemented as a REST API taking a record in JSON as the input and returning a prediction result. The time taken to complete each prediction can take from milliseconds to seconds. The response for each API call can be complex to calculate because it requires multiple joins, for example, resolving user attributes from cookie id. Ensemble models might not meet low latency requirements for REST API calls."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Offline prediction:"]}," Typically scheduled as a daily batch job. It can achieve high throughput for predictions but it generally takes tens of minutes to several hours to complete each batch of prediction."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Incremental (mini-batch) prediction:"]}," Similar to batch prediction with the difference that tasks can be split into smaller batches. It can take seconds to minutes to complete each mini-batch of prediction. It can not be applied for strict real-time requirements but can be applied for semi-realtime scenarios where a few minutes' latency is acceptable."]}]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["When storing prediction results in a key-value store, it is possible to achieve milliseconds for the latency."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Implement incremental mini-batch predictions"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Consider the use case of assigning scores to new users, who requested a sales callback for an insurance product. However, the number of users is large and it is required to prioritize them by predicted LTV (Lifetime Value) scores."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Many new customers need to be processed during each five minutes batch and some existing customers may have additional records. So, it's expected to calculate new LTV scores for users who accessed in the last five minutes. Prediction need to be completed within five minutes before the next batch starts."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Assuming the prediction result is stored on ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["${predicted_table}"]},", you can retrieve the user details sorted by the latest LTV scores, using the following SQL query:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Get latest LTV score for each user"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql","header":{"controls":{"copy":{}}},"source":"WITH scores AS (\n  SELECT \n    userid,\n    td_last(time, score) as score # use the latest prediction score\n  FROM\n    ${predicted_table}\n  WHERE\n    score >= 0.7 # filter by score threshold\n  GROUP BY\n    userid\n)\nSELECT\n  l.userid, l.score,\n  r.user_name, r.user_age, r.user_email, r.user_tel_no\nFROM \n  scores l\n  JOIN user_info r ON (l.userid=r.userid)\nORDER BY\n  score DESC \n-- LIMIT 100  \n","lang":"sql"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The following workflow can be used for scheduling mini-batch prediction that appends LTV scores to users who accessed in the last 5 minutes."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A workflow to run mini-batch prediction"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"yaml","header":{"controls":{"copy":{}}},"source":"# predict every 5 min\ntimezone: Asia/Tokyo\nschedule:\n  cron>: */5 * * * * \n\n# List users who accessed in the last 5 minutes\n+list_users:\n  td>: queries/list_users.sql\n  store_last_results: true\n\n# Run mini-batch predictions\n+gluon_predict:\n  ml_predict>:\n    notebook: gluon_predict\n    model_name: ${model_name}\n    input_table: ${input_database}.${input_table} \n    output_table: ${output_database}.${output_table}\n    output_mode: append # append scores to output_table\n    rowid_column: userid\n    rowid_filter: ${td.last.results.users}  \n","lang":"yaml"},"children":[]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["It is assumed that the prediction model, specified in ${model_name}, is created on a monthly basis and a verified well-performing model is used for prediction."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["In \"",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["queries/list_users.sql"]}," \", the ids of users who accessed in the last 5 minutes can be listed as follows:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["queries/list_users.sql"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql","header":{"controls":{"copy":{}}},"source":"SELECT \n  -- comma separated list of users: '111','222'\n  array_join(transform(array_agg(DISTINCT userid), x -> '''' || x || ''''), ',')  as users\nFROM\n  session\nWHERE\n  time >= TD_TIME_ADD(TD_SCHEDULED_TIME(), '-5m', 'JST') -- access in the last 5 minutes  \n","lang":"sql"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The \"",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["rowid_column"]}," \" and \"",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["rowid_filter"]}," \" are used to issue the following SQL query and only the matched rows are used for prediction."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"sql","header":{"controls":{"copy":{}}},"source":"SELECT * FROM ${input_database}.{input_table}\nWHERE {rowid_column} in ({rowid_filter})  \n","lang":"sql"},"children":[]}]},"headings":[{"value":"Incremental Mini Batch Prediction","id":"incremental-mini-batch-prediction","depth":1}],"frontmatter":{"seo":{"title":"Incremental Mini Batch Prediction"}},"lastModified":"2025-11-22T02:43:27.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/products/customer-data-platform/machine-learning/automl/advanced-topics/incremental-mini-batch-prediction","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}