Confiz Logo
Let's talk
Search
Close this search box.

Forecast modeling in production: A survival analysis

September 5, 2022

Subscribe to the newsletter

Popular Posts

With the evolution of technology, businesses are increasingly adopting machine learning and data analytics to derive solutions. According to Forbes, 50% of all enterprises planned to invest more in ML and AI in 2021.  

With this shift towards machine learning, data scientists often have to deal with challenges when models are in production. Last year, 38% of organizations reported that their data scientists spent 50% of their time in model deployment.  

One of the challenges that occur is drift in model predictions. Model drift typically refers to shifts in predictions made by the model; hence, what the model forecasts today differs from what it previously predicted. As a result, model predictions can become less reliable in the long run. This occurrence is particularly evident in time series forecast models, where some of the factors that can affect the accuracy of predictions are:

  • Shifts in consumer behavior
  • Changing monetary policies
  • Economic cycle changes
  • Competitor Activity

So, how can we respond to drifts in forecast models? We can use either a reactive or proactive approach to ensure model reliability. However, before delving deeper into the two different strategies, we have to familiarize ourselves with a couple of terms:

Model Reliability:

Within the machine learning community, the concept of reliability covers closely related ideas of uncertainty in model predictions.

Moreover, it also encompasses failures to generalize production data. Ultimately, the reliability of model predictions at a specific point in time refers to model reliability. You can trust a reliable model’s forecasts up to a certain period before you might want to replace it.

Model Availability:

Here, model availability means the availability of a model that satisfies production requirements for model reliability. Now that we’ve learned about the abovementioned terminology, we are going to go over the two different strategies:

The Reactive Approach

One solution to drifts in model predictions is to adopt a reactive approach and establish thresholds based on forecast-related business rules.

This way, a drift event can be detected if the difference between the actual time series and forecasts crosses a particular threshold point. Consequently, that would trigger retraining existing models or further analysis in an automated setting.  

Employing a reactive approach to drift events is reasonable if:

  1. Such events occur rarely
  2. Retraining expenses are low due to the low number of models in production
  3. Model availability does not have strict requirements

However, this approach might not be practicable in some industries because the number of models in production is high. Here, a proactive approach is more convenient.  

The Proactive Approach

In industries like retail or logistics, the number of time series that need to be modeled can easily cross thousands. Additionally, these time series may be closely related and belong to the same domain, material, or geography.  

In this case, responding to drift events at such a vast scale is unfeasible. That’s because responding with further analysis or model retraining can take up a lot of analysis and computational time.

We need to ensure that the model is reliable in production, has decent uptime, and does not drift away and trigger retraining. For this, we need a strategy that can foresee model drift in advance. Here, a proactive approach might be suitable. The objectives of this are to:

  • Be able to meet SLA requirements related to model availability.
  • Analyze these cases for root cause analysis in a pre-production setting.
  • Calculate computational requirements for retraining in advance (If the frequency of these events across models is known).

A Proactive approach can provide a reasonable estimate of these events even when the model is in a pre-production state.

How to analyze forecast model reliability with survival analysis?

An excellent approach to estimating reliability in time series forecast models is through survival analysis. You can create a survival model to predict model delay and perform a proactive analysis if:

  • Significant history is available on model performance
  • Numerous closely related models are available  

Proactive analysis helps you quantify both model availability under different scenarios and computational requirements for retraining. Moreover, it will help you pre-plan. Before we demonstrate how to use survival analysis to predict the kind of decay expected from a given set of time series, we’ll have a brief overview of how the survival model works.  

What do you need to know about Survival Analysis?

Models that predict the time to an event are called survival models. Survival models fall under a branch of statistics that deals with the analysis of time to an event, called survival analysis.

For example, the time before a loan default or time before an equipment failure are examples of survival models. (Loan default and equipment failure are the “events” in these scenarios.) You can model the survival probability distribution as follows:

Where,

T: waiting time until the occurrence of an event

S(t): Probability that waiting time T is > than some time t. Meaning, probability of survival till time t.  

From here, we can go to the concept of censorship. If the time the event occurs is unknown, it’s said to be censored. If the observation period of an experiment expires and the subject does not face an event (will face one in the future), it’s called right censored. Right censorship is the most relevant to our forecasting model reliability study.

‍

Survival modeling techniques are classified broadly into two techniques:

Parametric Models:

These models make assumptions about the form of the survival probability distribution. They can be good choices if their assumptions are backed with domain knowledge.

Non-Parametric Models:

These models empirically calculate the probability distribution from available data instead of making assumptions. The end goal for both models is to acquire a survival function that, given a particular point in time, would output the survival probability up to that point.

Using the M5 forecasting dataset as an example:

Now, we will take an example of reliability in a time series model problem and walk through its solution. We’ll use the M5 forecasting dataset in this example. The fifth edition of the M-series forecasting competitions, directed by Professor Spyros Makridakis, used this dataset. You may access the dataset here.

The data includes sales at different levels of the hierarchy, such as item, department, product categories, and store. In addition, it features retailers from the US States like California, Texas, and Wisconsin.

‍

iditem_iddept_idcat_idstore_idstate_idd_1d_2d_3d_4d_5d_6d_7d_8d_9d_10d_11d_12d_13d_14d_15d_16d_17d_18d_19d_20d_21d_22d_23d_24d_25d_26d_27d_28d_29d_30d_31d_32d_33d_34d_35d_36d_37d_38d_39d_40d_41d_42d_43d_44d_45d_46d_47d_48d_49d_50d_51d_52d_53d_54d_55d_56d_57d_58d_59d_60d_61d_62d_63d_64d_65d_66d_67d_68d_69d_70d_71d_72d_73d_74d_75d_76d_77d_78d_79d_80d_81d_82d_83d_84d_85d_86d_87d_88d_89d_90d_91d_92d_93d_94d_95d_96d_97d_98d_99d_100d_101d_102d_103d_104d_105d_106d_107d_108d_109d_110d_111d_112d_113d_114d_115d_116d_117d_118d_119d_120d_121d_122d_123d_124d_125d_126d_127d_128d_129d_130d_131d_132d_133d_134d_135d_136d_137d_138d_139d_140d_141d_142d_143d_144d_145d_146d_147d_148d_149d_150d_151d_152d_153d_154d_155d_156d_157d_158d_159d_160d_161d_162d_163d_164d_165d_166d_167d_168d_169d_170d_171d_172d_173d_174d_175d_176d_177d_178d_179d_180d_181d_182d_183d_184d_185d_186d_187d_188d_189d_190d_191d_192d_193d_194d_195d_196d_197d_198d_199d_200d_201d_202d_203d_204d_205d_206d_207d_208d_209d_210d_211d_212d_213d_214d_215d_216d_217d_218d_219d_220d_221d_222d_223d_224d_225d_226d_227d_228d_229d_230d_231d_232d_233d_234d_235d_236d_237d_238d_239d_240d_241d_242d_243d_244d_245d_246d_247d_248d_249d_250d_251d_252d_253d_254d_255d_256d_257d_258d_259d_260d_261d_262d_263d_264d_265d_266d_267d_268d_269d_270d_271d_272d_273d_274d_275d_276d_277d_278d_279d_280d_281d_282d_283d_284d_285d_286d_287d_288d_289d_290d_291d_292d_293d_294d_295d_296d_297d_298d_299d_300d_301d_302d_303d_304d_305d_306d_307d_308d_309d_310d_311d_312d_313d_314d_315d_316d_317d_318d_319d_320d_321d_322d_323d_324d_325d_326d_327d_328d_329d_330d_331d_332d_333d_334d_335d_336d_337d_338d_339d_340d_341d_342d_343d_344d_345d_346d_347d_348d_349d_350d_351d_352d_353d_354d_355d_356d_357d_358d_359d_360d_361d_362d_363d_364d_365d_366d_367d_368d_369d_370d_371d_372d_373d_374d_375d_376d_377d_378d_379d_380d_381d_382d_383d_384d_385d_386d_387d_388d_389d_390d_391d_392d_393d_394d_395d_396d_397d_398d_399d_400d_401d_402d_403d_404d_405d_406d_407d_408d_409d_410d_411d_412d_413d_414d_415d_416d_417d_418d_419d_420d_421d_422d_423d_424d_425d_426d_427d_428d_429d_430d_431d_432d_433d_434d_435d_436d_437d_438d_439d_440d_441d_442d_443d_444d_445d_446d_447d_448d_449d_450d_451d_452d_453d_454d_455d_456d_457d_458d_459d_460d_461d_462d_463d_464d_465d_466d_467d_468d_469d_470d_471d_472d_473d_474d_475d_476d_477d_478d_479d_480d_481d_482d_483d_484d_485d_486d_487d_488d_489d_490d_491d_492d_493d_494d_495d_496d_497d_498d_499d_500d_501d_502d_503d_504d_505d_506d_507d_508d_509d_510d_511d_512d_513d_514d_515d_516d_517d_518d_519d_520d_521d_522d_523d_524d_525d_526d_527d_528d_529d_530d_531d_532d_533d_534d_535d_536d_537d_538d_539d_540d_541d_542d_543d_544d_545d_546d_547d_548d_549d_550d_551d_552d_553d_554d_555d_556d_557d_558d_559d_560d_561d_562d_563d_564d_565d_566d_567d_568d_569d_570d_571d_572d_573d_574d_575d_576d_577d_578d_579d_580d_581d_582d_583d_584d_585d_586d_587d_588d_589d_590d_591d_592d_593d_594d_595d_596d_597d_598d_599d_600d_601d_602d_603d_604d_605d_606d_607d_608d_609d_610d_611d_612d_613d_614d_615d_616d_617d_618d_619d_620d_621d_622d_623d_624d_625d_626d_627d_628d_629d_630d_631d_632d_633d_634d_635d_636d_637d_638d_639d_640d_641d_642d_643d_644d_645d_646d_647d_648d_649d_650d_651d_652d_653d_654d_655d_656d_657d_658d_659d_660d_661d_662d_663d_664d_665d_666d_667d_668d_669d_670d_671d_672d_673d_674d_675d_676d_677d_678d_679d_680d_681d_682d_683d_684d_685d_686d_687d_688d_689d_690d_691d_692d_693d_694d_695d_696d_697d_698d_699d_700d_701d_702d_703d_704d_705d_706d_707d_708d_709d_710d_711d_712d_713d_714d_715d_716d_717d_718d_719d_720d_721d_722d_723d_724d_725d_726d_727d_728d_729d_730d_731d_732d_733d_734d_735d_736d_737d_738d_739d_740d_741d_742d_743d_744d_745d_746d_747d_748d_749d_750d_751d_752d_753d_754d_755d_756d_757d_758d_759d_760d_761d_762d_763d_764d_765d_766d_767d_768d_769d_770d_771d_772d_773d_774d_775d_776d_777d_778d_779d_780d_781d_782d_783d_784d_785d_786d_787d_788d_789d_790d_791d_792d_793d_794d_795d_796d_797d_798d_799d_800d_801d_802d_803d_804d_805d_806d_807d_808d_809d_810d_811d_812d_813d_814d_815d_816d_817d_818d_819d_820d_821d_822d_823d_824d_825d_826d_827d_828d_829d_830d_831d_832d_833d_834d_835d_836d_837d_838d_839d_840d_841d_842d_843d_844d_845d_846d_847d_848d_849d_850d_851d_852d_853d_854d_855d_856d_857d_858d_859d_860d_861d_862d_863d_864d_865d_866d_867d_868d_869d_870d_871d_872d_873d_874d_875d_876d_877d_878d_879d_880d_881d_882d_883d_884d_885d_886d_887d_888d_889d_890d_891d_892d_893d_894d_895d_896d_897d_898d_899d_900d_901d_902d_903d_904d_905d_906d_907d_908d_909d_910d_911d_912d_913d_914d_915d_916d_917d_918d_919d_920d_921d_922d_923d_924d_925d_926d_927d_928d_929d_930d_931d_932d_933d_934d_935d_936d_937d_938d_939d_940d_941d_942d_943d_944d_945d_946d_947d_948d_949d_950d_951d_952d_953d_954d_955d_956d_957d_958d_959d_960d_961d_962d_963d_964d_965d_966d_967d_968d_969d_970d_971d_972d_973d_974d_975d_976d_977d_978d_979d_980d_981d_982d_983d_984d_985d_986d_987d_988d_989d_990d_991d_992d_993d_994d_995d_996d_997d_998d_999d_1000d_1001d_1002d_1003d_1004d_1005d_1006d_1007d_1008d_1009d_1010d_1011d_1012d_1013d_1014d_1015d_1016d_1017d_1018d_1019d_1020d_1021d_1022d_1023d_1024d_1025d_1026d_1027d_1028d_1029d_1030d_1031d_1032d_1033d_1034d_1035d_1036d_1037d_1038d_1039d_1040d_1041d_1042d_1043d_1044d_1045d_1046d_1047d_1048d_1049d_1050d_1051d_1052d_1053d_1054d_1055d_1056d_1057d_1058d_1059d_1060d_1061d_1062d_1063d_1064d_1065d_1066d_1067d_1068d_1069d_1070d_1071d_1072d_1073d_1074d_1075d_1076d_1077d_1078d_1079d_1080d_1081d_1082d_1083d_1084d_1085d_1086d_1087d_1088d_1089d_1090d_1091d_1092d_1093d_1094d_1095d_1096d_1097d_1098d_1099d_1100d_1101d_1102d_1103d_1104d_1105d_1106d_1107d_1108d_1109d_1110d_1111d_1112d_1113d_1114d_1115d_1116d_1117d_1118d_1119d_1120d_1121d_1122d_1123d_1124d_1125d_1126d_1127d_1128d_1129d_1130d_1131d_1132d_1133d_1134d_1135d_1136d_1137d_1138d_1139d_1140d_1141d_1142d_1143d_1144d_1145d_1146d_1147d_1148d_1149d_1150d_1151d_1152d_1153d_1154d_1155d_1156d_1157d_1158d_1159d_1160d_1161d_1162d_1163d_1164d_1165d_1166d_1167d_1168d_1169d_1170d_1171d_1172d_1173d_1174d_1175d_1176d_1177d_1178d_1179d_1180d_1181d_1182d_1183d_1184d_1185d_1186d_1187d_1188d_1189d_1190d_1191d_1192d_1193d_1194d_1195d_1196d_1197d_1198d_1199d_1200d_1201d_1202d_1203d_1204d_1205d_1206d_1207d_1208d_1209d_1210d_1211d_1212d_1213d_1214d_1215d_1216d_1217d_1218d_1219d_1220d_1221d_1222d_1223d_1224d_1225d_1226d_1227d_1228d_1229d_1230d_1231d_1232d_1233d_1234d_1235d_1236d_1237d_1238d_1239d_1240d_1241d_1242d_1243d_1244d_1245d_1246d_1247d_1248d_1249d_1250d_1251d_1252d_1253d_1254d_1255d_1256d_1257d_1258d_1259d_1260d_1261d_1262d_1263d_1264d_1265d_1266d_1267d_1268d_1269d_1270d_1271d_1272d_1273d_1274d_1275d_1276d_1277d_1278d_1279d_1280d_1281d_1282d_1283d_1284d_1285d_1286d_1287d_1288d_1289d_1290d_1291d_1292d_1293d_1294d_1295d_1296d_1297d_1298d_1299d_1300d_1301d_1302d_1303d_1304d_1305d_1306d_1307d_1308d_1309d_1310d_1311d_1312d_1313d_1314d_1315d_1316d_1317d_1318d_1319d_1320d_1321d_1322d_1323d_1324d_1325d_1326d_1327d_1328d_1329d_1330d_1331d_1332d_1333d_1334d_1335d_1336d_1337d_1338d_1339d_1340d_1341d_1342d_1343d_1344d_1345d_1346d_1347d_1348d_1349d_1350d_1351d_1352d_1353d_1354d_1355d_1356d_1357d_1358d_1359d_1360d_1361d_1362d_1363d_1364d_1365d_1366d_1367d_1368d_1369d_1370d_1371d_1372d_1373d_1374d_1375d_1376d_1377d_1378d_1379d_1380d_1381d_1382d_1383d_1384d_1385d_1386d_1387d_1388d_1389d_1390d_1391d_1392d_1393d_1394d_1395d_1396d_1397d_1398d_1399d_1400d_1401d_1402d_1403d_1404d_1405d_1406d_1407d_1408d_1409d_1410d_1411d_1412d_1413d_1414d_1415d_1416d_1417d_1418d_1419d_1420d_1421d_1422d_1423d_1424d_1425d_1426d_1427d_1428d_1429d_1430d_1431d_1432d_1433d_1434d_1435d_1436d_1437d_1438d_1439d_1440d_1441d_1442d_1443d_1444d_1445d_1446d_1447d_1448d_1449d_1450d_1451d_1452d_1453d_1454d_1455d_1456d_1457d_1458d_1459d_1460d_1461d_1462d_1463d_1464d_1465d_1466d_1467d_1468d_1469d_1470d_1471d_1472d_1473d_1474d_1475d_1476d_1477d_1478d_1479d_1480d_1481d_1482d_1483d_1484d_1485d_1486d_1487d_1488d_1489d_1490d_1491d_1492d_1493d_1494d_1495d_1496d_1497d_1498d_1499d_1500d_1501d_1502d_1503d_1504d_1505d_1506d_1507d_1508d_1509d_1510d_1511d_1512d_1513d_1514d_1515d_1516d_1517d_1518d_1519d_1520d_1521d_1522d_1523d_1524d_1525d_1526d_1527d_1528d_1529d_1530d_1531d_1532d_1533d_1534d_1535d_1536d_1537d_1538d_1539d_1540d_1541d_1542d_1543d_1544d_1545d_1546d_1547d_1548d_1549d_1550d_1551d_1552d_1553d_1554d_1555d_1556d_1557d_1558d_1559d_1560d_1561d_1562d_1563d_1564d_1565d_1566d_1567d_1568d_1569d_1570d_1571d_1572d_1573d_1574d_1575d_1576d_1577d_1578d_1579d_1580d_1581d_1582d_1583d_1584d_1585d_1586d_1587d_1588d_1589d_1590d_1591d_1592d_1593d_1594d_1595d_1596d_1597d_1598d_1599d_1600d_1601d_1602d_1603d_1604d_1605d_1606d_1607d_1608d_1609d_1610d_1611d_1612d_1613d_1614d_1615d_1616d_1617d_1618d_1619d_1620d_1621d_1622d_1623d_1624d_1625d_1626d_1627d_1628d_1629d_1630d_1631d_1632d_1633d_1634d_1635d_1636d_1637d_1638d_1639d_1640d_1641d_1642d_1643d_1644d_1645d_1646d_1647d_1648d_1649d_1650d_1651d_1652d_1653d_1654d_1655d_1656d_1657d_1658d_1659d_1660d_1661d_1662d_1663d_1664d_1665d_1666d_1667d_1668d_1669d_1670d_1671d_1672d_1673d_1674d_1675d_1676d_1677d_1678d_1679d_1680d_1681d_1682d_1683d_1684d_1685d_1686d_1687d_1688d_1689d_1690d_1691d_1692d_1693d_1694d_1695d_1696d_1697d_1698d_1699d_1700d_1701d_1702d_1703d_1704d_1705d_1706d_1707d_1708d_1709d_1710d_1711d_1712d_1713d_1714d_1715d_1716d_1717d_1718d_1719d_1720d_1721d_1722d_1723d_1724d_1725d_1726d_1727d_1728d_1729d_1730d_1731d_1732d_1733d_1734d_1735d_1736d_1737d_1738d_1739d_1740d_1741d_1742d_1743d_1744d_1745d_1746d_1747d_1748d_1749d_1750d_1751d_1752d_1753d_1754d_1755d_1756d_1757d_1758d_1759d_1760d_1761d_1762d_1763d_1764d_1765d_1766d_1767d_1768d_1769d_1770d_1771d_1772d_1773d_1774d_1775d_1776d_1777d_1778d_1779d_1780d_1781d_1782d_1783d_1784d_1785d_1786d_1787d_1788d_1789d_1790d_1791d_1792d_1793d_1794d_1795d_1796d_1797d_1798d_1799d_1800d_1801d_1802d_1803d_1804d_1805d_1806d_1807d_1808d_1809d_1810d_1811d_1812d_1813d_1814d_1815d_1816d_1817d_1818d_1819d_1820d_1821d_1822d_1823d_1824d_1825d_1826d_1827d_1828d_1829d_1830d_1831d_1832d_1833d_1834d_1835d_1836d_1837d_1838d_1839d_1840d_1841d_1842d_1843d_1844d_1845d_1846d_1847d_1848d_1849d_1850d_1851d_1852d_1853d_1854d_1855d_1856d_1857d_1858d_1859d_1860d_1861d_1862d_1863d_1864d_1865d_1866d_1867d_1868d_1869d_1870d_1871d_1872d_1873d_1874d_1875d_1876d_1877d_1878d_1879d_1880d_1881d_1882d_1883d_1884d_1885d_1886d_1887d_1888d_1889d_1890d_1891d_1892d_1893d_1894d_1895d_1896d_1897d_1898d_1899d_1900d_1901d_1902d_1903d_1904d_1905d_1906d_1907d_1908d_1909d_1910d_1911d_1912d_1913d_1914d_1915d_1916d_1917d_1918d_1919d_1920d_1921d_1922d_1923d_1924d_1925d_1926d_1927d_1928d_1929d_1930d_1931d_1932d_1933d_1934d_1935d_1936d_1937d_1938d_1939d_1940d_1941
HOBBIES_1_001_CA_1_evaluationHOBBIES_1_001HOBBIES_1HOBBIESCA_1CA000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000020110000012101100010000000101002100020000010001012110000100120100101010101000010000022011000110111102311000001000021001121101011011030030112011010000000110000012001020000001010110000010000012000000120002120010001001000010000001010111020100000000000111010000011201000001010000100000010000011020010001010000101010010011201000000003122110020000101000110310000010111000020020001000004002100000100000401012200002111110010010001101200201310010010000010010110110011100000200003020000320200200003011101010024000002001001151100000000211020210101100021100103000100210100330000000111022200100130001101010011121110222100012001211101010010102102010001120000000000000000000000000000000000110000000001311011000112000102101100011131100201000200042100002101100012110020220001102011201000210112000000010101031101120000110000301000011101020000200001120000200111100000122010000121000001030121030001022100120101400500000021210001110011111000220014000011204010142020110100113000111313122011110000010423012000113011130110002035001102122102400003301
HOBBIES_1_002_CA_1_evaluationHOBBIES_1_002HOBBIES_1HOBBIESCA_1CA000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001001000001101000010001000000000010000000000010010000011110100000100100010110100000010010000000001000100000000111000000101010010100000001001000110000000000000000000100000110200020031101421000010000010000100000001000010011001000010000001100001001001000000000010000000000000000000000000000002001000000000100000000000011000110000000000000010001000001000010010010000000001001000000010001000001000000000000000010000000000000101001000000100000003110000000000000000011100000000010020001000000002211000001000000000000110000110000000100100002000021100021303132133000000010000000000100000110000001002010101000000001100000000200000111100001000000000000000010101201000000000001000000001010000000011000011100200000000100000000011001000010010101000010000001020010000201000000110010000000000011000100100011000010100011000000100000000000000000000200010100001010000000000000001000000100000000000010001000000000000000000021000201110101000000000000000010000000000000000000000010010000000000000100000000001000000100000000010000000000010001000000100000000100110100100000200001000010010010202001001010000202002000001001030000001110021100101100000010110100010002001101100001000001001100000110002010000000000100000000000000021101100000002000100002001010001010111102011010000001000100010100010010101010014023301000000000000001020000000000000000010000010000000001000000000000100100000100000000010000100000000010000000000000100000000010000010100100001000103010001000000101000000000110001011000000000001110100000002100000100110202101000211100010100010011000340000100000000100011001100000002100100000000300000000001021000110000110101100000101103000000010000021001102010211501035001000000000000100000100100000000010001000000100000001000100001000000100110010100000000000111110000000000000000000000100000100000001000000000121100000
HOBBIES_1_003_CA_1_evaluationHOBBIES_1_003HOBBIES_1HOBBIESCA_1CA000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000030200101110001100000002000001000000000002010001030101101200000001000010100001000000100000200100000000010000020001000100000000000010001010000000000000000100000000210000000000000001000010000000000000000000000000010101100010100000000001200000000000000000000000000000000000000000000000000000000000000000000000000000100000000001000000100000000010001010110010111000000001001001001000000310001111100000000110110000010000100010001000000100000000000010000010100100010000101010010101010020001001000100200001110000001010021000010200000001000011010000030000001001111011000000000101000000210301203100101000020101011010100012000101111000000201002000100100200000000202301312230110000231143212201520123012130111100000001010000000000000000100000001011111000101010000000000006112000110000100000011000000010001000001221211101110011021000021300101020002301
HOBBIES_1_004_CA_1_evaluationHOBBIES_1_004HOBBIES_1HOBBIESCA_1CA0000000000000000000000000000000000002000201000000201001110231000001001100100123020022002121112001013100003522111111002111200025600000000000212010200510013135130344001314002020142102136112123120003451001014631101451140001221162440002201131242132204110422512332211213213000312301020542124110122202301111051112042320211631322120000310621210000006330000220421011813012331121111201200000003282012025300310212212260201121300214311112350121202200000211012203000030110311611000000000014112123100111310110205101121300011110000120010001001110330120034010000212112100003022121121021251151012114150700312311421301132111151350112051021215220406300000000000010140000000000000000211100001000000000021120000000000000000000000000000000000000000006111102103023121110120103102126011160300143202224220121271010251610003403211252000023312310531121112012020237002123102131000364001120011001033000213000113302002000000000000013311014212123222500147000210421111050101312141401142114133030113410010244010354010103512431122121086321225502210311111024220134323253283016123120110143001226200301051324420141025110112220003285221124200502462013053031101240221401722016100111226101305214252421001112120325321121334142211411111214220212332204642112125221235100021132223025323311122531365241324571044141013136120303822002420024236203520541101612543234323203612133150210290022300032001330030045122005011000461044082412140471516550131150501163223131824002115902250711300210210024330223019204112381002113310212600203104011114282221112200134102121310120010320403110100457002014171108208231138410135230212167021038411122652101342222210130005325102100120102241311414220231029220010201032471013350000102500110741140005110001122112112110302140003311110331731001011002144302001330212470210552241003100031330041111331030133322243051332001102223212050100034001532201102102400032431230821225261035116432232100020542112306000101531000123013421413506600003121310254203010541013720012416400022421111104013026
HOBBIES_1_005_CA_1_evaluationHOBBIES_1_005HOBBIES_1HOBBIESCA_1CA000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000086032353100102240031112200000037110000000010000000001000000000000000000332000000020624200324100001112000000002679476342015220000001330020011300000114010031211110110010003400301202321401010110012000220000113360022143280000000010210250301100020210000000000000022331200300040100000000000000000000000000000000000000100000000000000000000000000000000000000000000000002213020000253223131120122252112201410010101121201101201011012000200100102011014210000023011003400200013201022020000000000000000204002310021513011001212211102200200212003113220041172301000013021311202070002011111010120213524001352011204110135220010013200410600022401121202000002310200214002122100000202102230220120211014111200000001001123111302111121000010000001101011541023131000000000102121112201010001000041300010320213512010110240000016100111002320000020220115001223010233010134111021220310220133111003001002310301111141000111000120311000011000010010321201100401101101201000300141022500120001100010000003100002011200422120131000020020210111022210220000101200210000323010011100111000120311010000020110230011101200003100010211220114000121000002100120131130500001021102000001020220101000101010400020102212000100002320001011300022020011030000021500020000000023110111000012200100033111019600012610011012010022411032720010110031002310100542021313210024232200103003011100011000121111111020021041210011510120111013311410002120211122101120142022131120201001014001020132112233001102021202342000000000000200000031000021100000512101110021010011003311032233210012110211002100000110111212102210333412020520010310102100131104021240100312201104004022212111402120010112231023011403212122120112020004212000002100120121212330315321234001001001000020030012201000100300110310412000112005222100030003111121001021103112110322231000010440140101011201121101122241023103231132322220002100210

view rawM5_sales_head.csv hosted with ❤ by GitHub

‍

Methodology

To undertake this study, we will take some special considerations in two different pipeline phases:

  • The pre-processing stage
  • The evaluation metric generation

The experiment design is as follows:

The Pre-processing Stage

Preprocessing involves filtering the stores and departments that interest us depending on the configuration. The sales columns are then melted (converted from wide to long format) into a single column. The sales data is presented at an item level. Therefore, you can sum this up at each store department level. As a result, you can acquire store-level sales data for each department.

Modeling

Prophet model, a well-known time series forecasting model, built by Facebook research, is created for every store department. These models are developed using default parameters, with a one-year test forecast horizon.  

Survival Analysis Design

‍

Here, we will walk through all the steps required to perform survival modeling. The two utilities for this are:

  1. The first tool analyses the test set and computes MAPE (Mean absolute percentage error), a common metric to assess time series forecasts, over an extending window from week 4 to week 26. (6 months).  

By assuming that the time to event for a forecasting model in production is the period before its performance deteriorates beyond a specific threshold, we can use survival analysis to model this waiting period and proactively take measures to ensure model dependability.

We can calculate the time before a model degrades, decays, or drifts by applying a metric like expanding window MAPE, which generates evaluation scores for a model against time. Then, we can model that time in turn.

  1. The first utility provides a time-based evaluation metric on the test set. In contrast, the second utility sets a threshold for the first week when performance begins to decline and records this as an event.

The configuration’s MAPE threshold is 10. Simply put, we mark an event when the expanding MAPE crosses ten in the first week. For store-department models, where the threshold isn’t crossed by the end of the research, 26 weeks are designated as censored.

Using The Kaplan Meier Estimator

Kaplan Meier Estimator is typically the first baseline model for survival analysis. It is a non-parametric estimator of the survival function. Let’s assume that there is no domain knowledge and covariates to keep us informed here. In this scenario, we are going to use this estimator for illustration.

Estimator Formula:

Where,

dᵢ: Events that happened at tᵢ

nᵢ: Total subjects that did not face an event up to tᵢ

Now, we accumulate all event observations and censors from the store-department models and build Kaplan-Meier models for every department.

Results:

As a result of fitting the Kaplan Meier Estimator, we get the survival function that is plotted below:

‍

From this figure, we can acquire the survival probability of models belonging to the Hobbies and Food department for every week in the study.  

Interpretation of the results:

Probability of survival at the start of the study: 1.0

Median Survival Time (Time where the probability of survival is 0.5): 4 weeks for both departments.

We can ensure model reliability by taking preemptive measures like scheduling retraining and conducting analysis without waiting for a performance event in production. To do this, we need the  

An example setup could use the 95th percentile survival time of 2 weeks and retrain all store-level models in a specific department before this limit is met in production. As a result, it can ensure that the model degradation is below the threshold with the given probability.

Concluding Remarks:

Once we take proactive steps like the ones mentioned using survival analysis, we can facilitate model availability. We can also use this method with models from various categories and across modeling approaches for cases other than retail and logistics.

Moreover, as this approach involves new metrics for generating time-to-failure events, they open up many opportunities for studying model reliability in production. Furthermore, these metrics also serve as individual evaluation metrics, which can be valuable for time series forecasting models.  

While this article is specific to issues in forecast modeling, data scientists can apply the same principle to other modeling approaches, given the appropriate evaluation metrics.

You may also like

Dynamics 365 Partner

Dynamics 365 partners in the USA: Top 5 picks for 2025 
Joined Record Size

Fixing the record size limit error in Dynamics 365 Finance & Operations: A quick guide
Business intelligence vs data analytics

Business Intelligence vs Data Analytics: How both power success in modern enterprises