Add Friedland Chapter 7 Jupyter notebook

[salexanian]

Summary of Changes

This PR replaces the Chapter 7 .rst documentation with a Jupyter notebook.

Current progress:

-Migrated Exhibits I and II.
-Added Exhibit III.
-Updated the supporting data.

Remaining work includes:

-Exhibit IV.
-Assertions against Friedland.
-Formatting and notebook cleanup.

Related GitHub Issue(s)

#588

Additional Context for Reviewers

I will post a follow-up comment with several implementation observations and questions.

Checklist

• I passed tests locally for both code (uv run pytest) and documentation changes (uv run --directory docs jb build . --builder=custom --custom-builder=doctest)

---

Note

Low Risk
Documentation navigation only; no runtime or library behavior changes in this diff.

Overview
Updates docs/_toc.yml so Friedland Chapter 7 is presented as two sibling sections instead of a single entry.

The existing friedland/chapter_7.rst page is labeled Chapter 7 - Part I, and a new friedland/chapter_7_part_2.ipynb page is added to the Friedland section as Chapter 7 - Part II.

^{Reviewed by Cursor Bugbot for commit 26b2665. Bugbot is set up for automated code reviews on this repo. Configure here.}

[salexanian]

I have been away from this issue for some time so please let me know if what I am doing is now obsolete/no longer required. If it is not, I hope to continue to help you and @priyam0k put this issue to rest.
[friedland_ch7_USPP.xlsx](https://github.com/user-

@salexanian your workbook isn't wasted. i think the useful role for it is as the source for the hardcoded numbers in the asserts, especially where friedland's printed figures are rounded. the reconciliation still has to happen in the notebook, but the numbers have to come from somewhere trustworthy.
i went through the same USPP scenarios for chapter 10's exhibit III, so a few things that saved me time:

• use IPython.display.display() on the triangle or dataframe instead of print
• display the rounded values like the text, but project ultimates off the unrounded estimator, otherwise steady state stops reconciling exactly
• one cell at the bottom with np.isclose / np.allclose per exhibit. i've been using rtol 5e-3 for totals and atol 1e-3 for cdfs and ratios

one thing i'd value your read on. steady_state and increasing_claim reconcile to the text for me, but increasing_case and increasing_claim_case don't. i think those two CSVs need corrections. if your workbook reproduces the text for them, that settles it and i'll raise the data fix.
also, so we don't write the same cells twice: i have exhibit III and IV drafted in the old rst form. since the direction is notebooks now, i can hand those over as raw material or port them myself, whichever you prefer. your call.

Hi @priyam0k - thank you for the helpful advice! I will check my code and ensure it follows what you listed above - I will send you and @henrydingliu my notebook to ensure that I am conforming with your approach. Re: the notebook - at least I am getting some Exam 5 practice out of it!

I will re-check my results against Friedland, and will look at your reconciliation as well for Exhibit III - I will add the asserts to my notebook and send you both the results. I recall there being some small discrepancies as well, but I will give you the details shortly. I would also be glad to use your code for Exhibit IV if you have it.

Hi @henrydingliu and @priyam0k , I attach my work-in-progress notebook for Ch. 7. It currently contains the following:

1. Exhibits I and II as prepared by @henrydingliu in his .rst, adapted to the Jupyter notebook.
2. Exhibit III from my existing notebook.
3. An empty section for Exhibit IV, which I will add using @priyam0k 's Chapter 10 code.

What remains to be done (which I am continuing to work on) is the following:

1. Implement asserts comparing the generated results to those in Friedland. I will follow @priyam0k 's error tolerance conventions for these.
2. Fix inconsistencies in the rendering of headings, subheadings, etc. across the Exhibits.
3. Fix the formatting of the numbers/significant digits in the Exhibit III output (from my notebook).
4. Add Exhibit IV.
5. Simplify my Exhibit III code to conform to @henrydingliu and @priyam0k presentation style in their notebook sections.

I have the following questions/observations to make based on my work so far:

1. There is a warning "UserWarning: Some exclusions have been ignored. At least 1 (use preserve = ...) link ratio(s) is required for development estimation." which I have been able to trace to the _drop_n and _drop_x functions in the DevelopmentBase class in base.py. Essentially, the warning is intended to remind a user to set the preserve parameter to ensure that the minimum intended number of ldf values are retained after dropping extreme values. Unfortunately, it appears that this warning flag will appear when a user sets preserve=1, which I think is a bug traceable the if statement in this code:

            if preserve == 1:
                warning = (
                    "Some exclusions have been ignored. At least "
                    + str(preserve)
                    + " (use preserve = ...)"
                    + " link ratio(s) is required for development estimation."
                )
            else:
                warning = (
                    "Some exclusions have been ignored. At least "
                    + str(preserve)
                    + " link ratio(s) is required for development estimation."
                )
            warnings.warn(warning)

If you agree that this is a bug or that the issue should be looked at further, I would be happy to take the appropriate steps. I had to supress this warning for the notebook to properly display some of the code without the warning.

2. Where we define functions that generate tables and perform all the associated formatting, should this part of the code be visible in the docs, or should these cells be hidden? If they are to be shown, should we be developing a standard way of showing everything, formatting tables, etc.? I will look into this issue myself as I complete the notebook, but wanted to flag this.

3. @henrydingliu had a helpful section in Exhibit I showing the unrounded and rounded ldfs. Unfortunately, when I moved the code to Jupyter from the .rst, the unrounded output now automatically rounds due to the following code in display.py, which reads as follows:

@staticmethod
    def _get_format_str(data: DataFrame) -> str:
        """
        Returns a numerical format string based on the magnitude of the mean absolute value of the values in the
        supplied DataFrame.

        Returns
        -------
        str
        """
        if np.all(np.isnan(data)):
            return ""
        elif np.nanmean(abs(data)) < 10:
            return "{0:,.4f}"
        elif np.nanmean(abs(data)) < 1000:
            return "{0:,.2f}"
        else:
            return "{:,.0f}"

Essentially, the default setting in Chainladder is to round HTML-rendered output to a prespecified number of decimal places for presentational quality depending on the average magnitude of the numbers (so based on the above the ldfs automatically round to 2 decimal places when rendered as HTML (which is what I think the notebook approach is doing). There are likely good reasons for the default display format above, but it does interfere with rendering of unrounded values in some circumstances (such as in the Jupyter notebook) and required a manual override. I raising this in case the default display code above needs to be reassessed.

4. I am trying to add some explanation of the sheets in Exhibit III (their purpose, context etc.), but if this is not required/needed (you can see some of the wording I put into Exhibit III so far), I can remove it or simplify it.

5. If I have done anything incorrectly in the notebook or should be following a specific protocol, please let me know and I will adjust accordingly.

I will send an update in the next couple of days, but in the meantime the work continues.

[github-actions]

Pyright Type Completeness

View the full pyright --verifytypes output for this commit

Project (full chainladder package, at this PR's head): 15.1% of exported symbols fully typed (201 / 1328)

	Known	Ambiguous	Unknown	Total
Project (head)	201	111	1016	1328

Other symbols referenced but not exported by chainladder: 13

	Known	Ambiguous	Unknown	Total
Other (head)	3	1	9	13

Symbols without documentation:

• Functions without docstring: 321
• Functions without default param: 0
• Classes without docstring: 10

Patch (exported symbols added or changed by this PR): no exported symbol type-completeness changes detected.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.26%. Comparing base (6ca44ed) to head (26b2665).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1189      +/-   ##
==========================================
- Coverage   91.48%   91.26%   -0.22%     
==========================================
  Files          91       91              
  Lines        5552     5401     -151     
  Branches      736      691      -45     
==========================================
- Hits         5079     4929     -150     
  Misses        338      338              
+ Partials      135      134       -1     

Flag	Coverage Δ
unittests	91.26% <ø> (-0.22%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[henrydingliu]

one general comment first. if you are still working through things, keep the PR as a draft

There is a warning "UserWarning: Some exclusions have been ignored. At least 1 (use preserve = ...) link ratio(s) is required for development estimation."

it's an annoyance for sure. but we also value the transparency. just leave it there for the time being

Where we define functions that generate tables and perform all the associated formatting, should this part of the code be visible in the docs, or should these cells be hidden?

keep the code visible. as for standardization, i think having a bit of variety actually gives our repo more character, i.e. we have a vibrant group of collaborators with different coding styles. there's some stuff we can standardize after we are through all the chapters. but not a priority for the time being.

had a helpful section in Exhibit I showing the unrounded and rounded ldfs. Unfortunately, when I moved the code to Jupyter from the .rst, the unrounded output now automatically rounds due to the following code in display.py, which reads as follows:

just leave the rst for now. you can change the rst title to part I and have both parts on the toc

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e809abb. Configure here.}

[priyam0k]

great progress on this @salexanian! since you're actively working through exhibit iv and the assertions, feel free to convert this to a draft PR so you don't feel rushed by review notifications. tag us whenever it's ready for final review!

[salexanian]

great progress on this @salexanian! since you're actively working through exhibit iv and the assertions, feel free to convert this to a draft PR so you don't feel rushed by review notifications. tag us whenever it's ready for final review!

Thank you @priyam0k for this advice - I actually didn't know that I could that, and it would be very helpful to have it in draft as I complete Chapter 7. I will try and convert it to draft and might circle back to you if I hit a roadblock.

[salexanian]

@priyam0k , @henrydingliu finally finished Part II of Chapter 7. Asserts are done to reconcile with Friedland and all pass. Unfortunately, due to rounding errors, there is a $2.03 discrepancy between an entry in Friedland's and my versions of Exhibit III, Sheets 10/11 that required manual adjustment of the tolerance. The asserts are stored in a new file, friedland_assertions.py and the Friedland values are stored in serialized form in a .json file.

[henrydingliu]

@salexanian thanks for putting in so much work! could you please follow the convention as everyone else and put the entire assert source code directly into the notebook, without having to load other modules or jsons?

[henrydingliu]

@salexanian thanks for putting in so much work! could you please follow the convention as everyone else and put the entire assert source code directly into the notebook, without having to load other modules or jsons?

btw we don't have to reconcile every single triangle and every single column. the key is reconciling estimated values.

[salexanian]

@salexanian thanks for putting in so much work! could you please follow the convention as everyone else and put the entire assert source code directly into the notebook, without having to load other modules or jsons?

btw we don't have to reconcile every single triangle and every single column. the key is reconciling estimated values.

No problem, @henrydingliu . I will make those changes and circle back shortly.

[salexanian]

@salexanian thanks for putting in so much work! could you please follow the convention as everyone else and put the entire assert source code directly into the notebook, without having to load other modules or jsons?

btw we don't have to reconcile every single triangle and every single column. the key is reconciling estimated values.

No problem, @henrydingliu . I will make those changes and circle back shortly.

@henrydingliu, @priyam0k - it is done.

[priyam0k]

In exhibit IV sheet 6 (steady state, change in product mix), a few cells render as 1, -1, and -0 instead of 0. this is just float precision passing through {:,.0f} on numbers. adding .round() to intermediate calculated columns in Ex4Sht6 cleans this up nicely

[salexanian]

In exhibit IV sheet 6 (steady state, change in product mix), a few cells render as 1, -1, and -0 instead of 0. this is just float precision passing through `{:,.0f}` on numbers. adding `.round()` to intermediate calculated columns in Ex4Sht6 cleans this up nicely

Hi @priyam0k - I tried to round but no luck. Looking more closely at Exhibit IV, Sheet 6 in the Friedland text, note the following (in the upper steady state portion of the table):

1. the total of column 3 is incorrect (it should be 17,472,204, not 17,472,205).
2. in year 2001, the value in column 5 is incorrect (the difference of the values in columns 3 and 4 should be 25,357, not 25,358).
3. in year 2002, the value in column 5 is again incorrect (the difference of the values in columns 3 and 4 should be 35,887, not 35,886).

In the lower changing product mix portion, I noted the following:

4. in year 2005, the value in column 5 is again incorrect (the difference of the values in columns 3 and 4 should be 242,110, not 242,111).
5. the total of column 3 is incorrect (it should be 20,067,180, not 20,067,179).

I have not looked at every entry, but the concerned me, particlarly given that columns 3 and 4 are not calculated columns (they are the data we are relying on). My guess is either Friedland's tables are incorrect (i.e. there are errors in her calculations), or else we do not have the raw data (i.e. the real data is of greater than integer precision and we only have integer-rounded values). What do you think about this?

I guess if the point of these exhibits is really to ensure that no code updates to Chainladder cause it to deviate from the numeric calculations it currently makes (i.e. we are interested in enforcing relative accuracy rather than absolute accuracy), it does not matter and we could adjust figures close to zero, down to zero to conform to the exhibits.

Please let me know your thoughts on this - I am uncertain what to do next.

[henrydingliu]

@salexanian thanks for the blazing fast revision!

a couple of other comments

• can you unhide the exhibit 3 functions?
• you can leverage cl.model_diagnostics() to massively simplify the amount of pandas wrangling in your functions. see this example from chapter 11

[priyam0k]

Hi @priyam0k - I tried to round but no luck. Looking more closely at Exhibit IV, Sheet 6 in the Friedland text, note the following (in the upper steady state portion of the table):

@salexanian friedland's textbook has known $1-$2 rounding errors from intermediate excel rounding when written.

in this repo, Dynamically calculating values via chainladder-python is our main priority. we never hardcode figures to match textbook typos.

using atol=3 in assert np.allclose() is our standard approach across docs for these rounding differences, so your assertion setup is good as-is!

@henrydingliu if u would like to add anything on this?

[henrydingliu]

i try to only use 1e-n as atol. using an arbitrary number like 3 just because that's the largest diff we have seems too goal-seeky. i'd recommend using rtol = 1e-n in those instances.

[salexanian]

@salexanian thanks for the blazing fast revision!

a couple of other comments

• can you unhide the exhibit 3 functions?
• you can leverage cl.model_diagnostics() to massively simplify the amount of pandas wrangling in your functions. see this example from chapter 11

No problem, @henrydingliu - I will make these changes. For the Ex. 3 functions, I think I forgot to remove the metatag (I believe it is called hide-input from the notebook's .json file - I will rectify it.

[salexanian]

i try to only use 1e-n as atol. using an arbitrary number like 3 just because that's the largest diff we have seems too goal-seeky. i'd recommend using rtol = 1e-n in those instances.

No problem, @henrydingliu - I recall using an rtol in the 1e-5 range was enough due to the magnitude of the dollar values - I'll try this approach and send you and @priyam0k my results.

[salexanian]

@salexanian thanks for the blazing fast revision!

a couple of other comments

• can you unhide the exhibit 3 functions?
• you can leverage cl.model_diagnostics() to massively simplify the amount of pandas wrangling in your functions. see this example from chapter 11

@henrydingliu , I have unhidden the Ex3 functions and hid the assert blocks as per your original request (in issue #588). I will now study the cl.model_diagnostics() function and then attempt to leverage it in the Ch. 7 Pt 2 code.