See the code lines starting at 2020. It looks like if result is still SCAN_SUCCEED, then page has iterated all the way around the loop and has reached the dummy list head. In that case, page->index on line 2038 is not likely to be a valid reference. julia ---------- Forwarded message ---------- Date: Sun, 13 Nov 2022 20:00:09 +0800 From: kernel test robot To: oe-kbuild@lists.linux.dev Cc: lkp@intel.com, Julia Lawall Subject: [ammarfaizi2-block:akpm/mm/mm-unstable 137/175] mm/khugepaged.c:2038:27-31: ERROR: invalid reference to the index variable of the iterator on line 2026 BCC: lkp@intel.com CC: oe-kbuild-all@lists.linux.dev CC: Ammar Faizi CC: "GNU/Weeb Mailing List" TO: Jiaqi Yan CC: Andrew Morton CC: Linux Memory Management List tree: https://github.com/ammarfaizi2/linux-block akpm/mm/mm-unstable head: 57832af0c4af4dc0da3f24703ebefc0ef83b0e1b commit: 180b0b616c523c240a8635478e563cdfccf493bb [137/175] mm/khugepaged: recover from poisoned file-backed memory :::::: branch date: 2 days ago :::::: commit date: 2 days ago config: x86_64-randconfig-c002 compiler: gcc-11 (Debian 11.3.0-8) 11.3.0 If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot | Reported-by: Julia Lawall cocci warnings: (new ones prefixed by >>) >> mm/khugepaged.c:2038:27-31: ERROR: invalid reference to the index variable of the iterator on line 2026 mm/khugepaged.c:2096:21-25: ERROR: invalid reference to the index variable of the iterator on line 2026 vim +2038 mm/khugepaged.c f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1737 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1738 /** 99cb0dbd47a15d Song Liu 2019-09-23 1739 * collapse_file - collapse filemap/tmpfs/shmem pages into huge one. f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1740 * 336e6b53d99ae3 Alex Shi 2020-12-14 1741 * @mm: process address space where collapse happens 34488399fa08fa Zach O'Keefe 2022-09-22 1742 * @addr: virtual collapse start address 336e6b53d99ae3 Alex Shi 2020-12-14 1743 * @file: file that collapse on 336e6b53d99ae3 Alex Shi 2020-12-14 1744 * @start: collapse start address 9710a78ab2aed0 Zach O'Keefe 2022-07-06 1745 * @cc: collapse context and scratchpad 336e6b53d99ae3 Alex Shi 2020-12-14 1746 * f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1747 * Basic scheme is simple, details are more complex: 87c460a0bded56 Hugh Dickins 2018-11-30 1748 * - allocate and lock a new huge page; 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1749 * - scan page cache replacing old pages with the new one 99cb0dbd47a15d Song Liu 2019-09-23 1750 * + swap/gup in pages if necessary; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1751 * + fill in gaps; 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1752 * + keep old pages around in case rollback is required; 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1753 * - if replacing succeeds: f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1754 * + copy data over; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1755 * + free old pages; 87c460a0bded56 Hugh Dickins 2018-11-30 1756 * + unlock huge page; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1757 * - if replacing failed; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1758 * + put all pages back and unfreeze them; 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1759 * + restore gaps in the page cache; 87c460a0bded56 Hugh Dickins 2018-11-30 1760 * + unlock and free huge page; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1761 */ 34488399fa08fa Zach O'Keefe 2022-09-22 1762 static int collapse_file(struct mm_struct *mm, unsigned long addr, 579c571e2efdb8 Song Liu 2019-09-23 1763 struct file *file, pgoff_t start, 34488399fa08fa Zach O'Keefe 2022-09-22 1764 struct collapse_control *cc) f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1765 { 579c571e2efdb8 Song Liu 2019-09-23 1766 struct address_space *mapping = file->f_mapping; 180b0b616c523c Jiaqi Yan 2022-11-06 1767 struct page *hpage, *page, *tmp; 95c66f836a4276 Gautam Menghani 2022-10-26 1768 pgoff_t index = 0, end = start + HPAGE_PMD_NR; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1769 LIST_HEAD(pagelist); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1770 XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1771 int nr_none = 0, result = SCAN_SUCCEED; 99cb0dbd47a15d Song Liu 2019-09-23 1772 bool is_shmem = shmem_file(file); 95c66f836a4276 Gautam Menghani 2022-10-26 1773 int nr = 0; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1774 99cb0dbd47a15d Song Liu 2019-09-23 1775 VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1776 VM_BUG_ON(start & (HPAGE_PMD_NR - 1)); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1777 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1778 result = alloc_charge_hpage(&hpage, mm, cc); 9710a78ab2aed0 Zach O'Keefe 2022-07-06 1779 if (result != SCAN_SUCCEED) f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1780 goto out; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1781 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1782) /* 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1783) * Ensure we have slots for all the pages in the range. This is 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1784) * almost certainly a no-op because most of the pages must be present 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1785) */ 95feeabb77149f Hugh Dickins 2018-11-30 1786 do { 95feeabb77149f Hugh Dickins 2018-11-30 1787 xas_lock_irq(&xas); 95feeabb77149f Hugh Dickins 2018-11-30 1788 xas_create_range(&xas); 95feeabb77149f Hugh Dickins 2018-11-30 1789 if (!xas_error(&xas)) 95feeabb77149f Hugh Dickins 2018-11-30 1790 break; 95feeabb77149f Hugh Dickins 2018-11-30 1791 xas_unlock_irq(&xas); 95feeabb77149f Hugh Dickins 2018-11-30 1792 if (!xas_nomem(&xas, GFP_KERNEL)) { 95feeabb77149f Hugh Dickins 2018-11-30 1793 result = SCAN_FAIL; 95feeabb77149f Hugh Dickins 2018-11-30 1794 goto out; 95feeabb77149f Hugh Dickins 2018-11-30 1795 } 95feeabb77149f Hugh Dickins 2018-11-30 1796 } while (1); 95feeabb77149f Hugh Dickins 2018-11-30 1797 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1798 __SetPageLocked(hpage); 99cb0dbd47a15d Song Liu 2019-09-23 1799 if (is_shmem) 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1800 __SetPageSwapBacked(hpage); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1801 hpage->index = start; 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1802 hpage->mapping = mapping; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1803 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1804 /* 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1805 * At this point the hpage is locked and not up-to-date. 87c460a0bded56 Hugh Dickins 2018-11-30 1806 * It's safe to insert it into the page cache, because nobody would 87c460a0bded56 Hugh Dickins 2018-11-30 1807 * be able to map it or use it in another way until we unlock it. f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1808 */ f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1809 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1810 xas_set(&xas, start); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1811 for (index = start; index < end; index++) { 180b0b616c523c Jiaqi Yan 2022-11-06 1812 page = xas_next(&xas); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1813 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1814 VM_BUG_ON(index != xas.xa_index); 99cb0dbd47a15d Song Liu 2019-09-23 1815 if (is_shmem) { 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1816 if (!page) { 701270fa193aad Hugh Dickins 2018-11-30 1817 /* 99cb0dbd47a15d Song Liu 2019-09-23 1818 * Stop if extent has been truncated or 99cb0dbd47a15d Song Liu 2019-09-23 1819 * hole-punched, and is now completely 99cb0dbd47a15d Song Liu 2019-09-23 1820 * empty. 701270fa193aad Hugh Dickins 2018-11-30 1821 */ 701270fa193aad Hugh Dickins 2018-11-30 1822 if (index == start) { 701270fa193aad Hugh Dickins 2018-11-30 1823 if (!xas_next_entry(&xas, end - 1)) { 701270fa193aad Hugh Dickins 2018-11-30 1824 result = SCAN_TRUNCATED; 042a30824871fa Hugh Dickins 2018-11-30 1825 goto xa_locked; 701270fa193aad Hugh Dickins 2018-11-30 1826 } 701270fa193aad Hugh Dickins 2018-11-30 1827 xas_set(&xas, index); 701270fa193aad Hugh Dickins 2018-11-30 1828 } 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1829 if (!shmem_charge(mapping->host, 1)) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1830 result = SCAN_FAIL; 042a30824871fa Hugh Dickins 2018-11-30 1831 goto xa_locked; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1832 } 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1833 xas_store(&xas, hpage); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1834 nr_none++; 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1835 continue; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1836 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1837 3159f943aafdba Matthew Wilcox 2017-11-03 1838 if (xa_is_value(page) || !PageUptodate(page)) { 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1839) struct folio *folio; 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1840) 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1841 xas_unlock_irq(&xas); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1842 /* swap in or instantiate fallocated page */ 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1843) if (shmem_get_folio(mapping->host, index, 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1844) &folio, SGP_NOALLOC)) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1845 result = SCAN_FAIL; 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1846 goto xa_unlocked; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1847 } 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1848) page = folio_file_page(folio, index); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1849 } else if (trylock_page(page)) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1850 get_page(page); 042a30824871fa Hugh Dickins 2018-11-30 1851 xas_unlock_irq(&xas); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1852 } else { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1853 result = SCAN_PAGE_LOCK; 042a30824871fa Hugh Dickins 2018-11-30 1854 goto xa_locked; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1855 } 99cb0dbd47a15d Song Liu 2019-09-23 1856 } else { /* !is_shmem */ 99cb0dbd47a15d Song Liu 2019-09-23 1857 if (!page || xa_is_value(page)) { 99cb0dbd47a15d Song Liu 2019-09-23 1858 xas_unlock_irq(&xas); 99cb0dbd47a15d Song Liu 2019-09-23 1859 page_cache_sync_readahead(mapping, &file->f_ra, 99cb0dbd47a15d Song Liu 2019-09-23 1860 file, index, e5a59d308f52bb David Howells 2020-09-04 1861 end - index); 99cb0dbd47a15d Song Liu 2019-09-23 1862 /* drain pagevecs to help isolate_lru_page() */ 99cb0dbd47a15d Song Liu 2019-09-23 1863 lru_add_drain(); 99cb0dbd47a15d Song Liu 2019-09-23 1864 page = find_lock_page(mapping, index); 99cb0dbd47a15d Song Liu 2019-09-23 1865 if (unlikely(page == NULL)) { 99cb0dbd47a15d Song Liu 2019-09-23 1866 result = SCAN_FAIL; 99cb0dbd47a15d Song Liu 2019-09-23 1867 goto xa_unlocked; 99cb0dbd47a15d Song Liu 2019-09-23 1868 } 75f360696ce9d8 Song Liu 2019-11-30 1869 } else if (PageDirty(page)) { 75f360696ce9d8 Song Liu 2019-11-30 1870 /* 75f360696ce9d8 Song Liu 2019-11-30 1871 * khugepaged only works on read-only fd, 75f360696ce9d8 Song Liu 2019-11-30 1872 * so this page is dirty because it hasn't 75f360696ce9d8 Song Liu 2019-11-30 1873 * been flushed since first write. There 75f360696ce9d8 Song Liu 2019-11-30 1874 * won't be new dirty pages. 75f360696ce9d8 Song Liu 2019-11-30 1875 * 75f360696ce9d8 Song Liu 2019-11-30 1876 * Trigger async flush here and hope the 75f360696ce9d8 Song Liu 2019-11-30 1877 * writeback is done when khugepaged 75f360696ce9d8 Song Liu 2019-11-30 1878 * revisits this page. 75f360696ce9d8 Song Liu 2019-11-30 1879 * 75f360696ce9d8 Song Liu 2019-11-30 1880 * This is a one-off situation. We are not 75f360696ce9d8 Song Liu 2019-11-30 1881 * forcing writeback in loop. 75f360696ce9d8 Song Liu 2019-11-30 1882 */ 75f360696ce9d8 Song Liu 2019-11-30 1883 xas_unlock_irq(&xas); 75f360696ce9d8 Song Liu 2019-11-30 1884 filemap_flush(mapping); 75f360696ce9d8 Song Liu 2019-11-30 1885 result = SCAN_FAIL; 75f360696ce9d8 Song Liu 2019-11-30 1886 goto xa_unlocked; 74c42e1baacf20 Rongwei Wang 2021-10-28 1887 } else if (PageWriteback(page)) { 74c42e1baacf20 Rongwei Wang 2021-10-28 1888 xas_unlock_irq(&xas); 74c42e1baacf20 Rongwei Wang 2021-10-28 1889 result = SCAN_FAIL; 74c42e1baacf20 Rongwei Wang 2021-10-28 1890 goto xa_unlocked; 99cb0dbd47a15d Song Liu 2019-09-23 1891 } else if (trylock_page(page)) { 99cb0dbd47a15d Song Liu 2019-09-23 1892 get_page(page); 99cb0dbd47a15d Song Liu 2019-09-23 1893 xas_unlock_irq(&xas); 99cb0dbd47a15d Song Liu 2019-09-23 1894 } else { 99cb0dbd47a15d Song Liu 2019-09-23 1895 result = SCAN_PAGE_LOCK; 99cb0dbd47a15d Song Liu 2019-09-23 1896 goto xa_locked; 99cb0dbd47a15d Song Liu 2019-09-23 1897 } 99cb0dbd47a15d Song Liu 2019-09-23 1898 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1899 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1900 /* b93b016313b3ba Matthew Wilcox 2018-04-10 1901 * The page must be locked, so we can drop the i_pages lock f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1902 * without racing with truncate. f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1903 */ f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1904 VM_BUG_ON_PAGE(!PageLocked(page), page); 4655e5e5f38726 Song Liu 2019-11-15 1905 4655e5e5f38726 Song Liu 2019-11-15 1906 /* make sure the page is up to date */ 4655e5e5f38726 Song Liu 2019-11-15 1907 if (unlikely(!PageUptodate(page))) { 4655e5e5f38726 Song Liu 2019-11-15 1908 result = SCAN_FAIL; 4655e5e5f38726 Song Liu 2019-11-15 1909 goto out_unlock; 4655e5e5f38726 Song Liu 2019-11-15 1910 } 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1911 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1912 /* 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1913 * If file was truncated then extended, or hole-punched, before 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1914 * we locked the first page, then a THP might be there already. 58ac9a8993a13e Zach O'Keefe 2022-09-22 1915 * This will be discovered on the first iteration. 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1916 */ 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1917 if (PageTransCompound(page)) { 58ac9a8993a13e Zach O'Keefe 2022-09-22 1918 struct page *head = compound_head(page); 58ac9a8993a13e Zach O'Keefe 2022-09-22 1919 58ac9a8993a13e Zach O'Keefe 2022-09-22 1920 result = compound_order(head) == HPAGE_PMD_ORDER && 58ac9a8993a13e Zach O'Keefe 2022-09-22 1921 head->index == start 58ac9a8993a13e Zach O'Keefe 2022-09-22 1922 /* Maybe PMD-mapped */ 58ac9a8993a13e Zach O'Keefe 2022-09-22 1923 ? SCAN_PTE_MAPPED_HUGEPAGE 58ac9a8993a13e Zach O'Keefe 2022-09-22 1924 : SCAN_PAGE_COMPOUND; 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1925 goto out_unlock; 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1926 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1927 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1928 if (page_mapping(page) != mapping) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1929 result = SCAN_TRUNCATED; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1930 goto out_unlock; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1931 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1932 74c42e1baacf20 Rongwei Wang 2021-10-28 1933 if (!is_shmem && (PageDirty(page) || 74c42e1baacf20 Rongwei Wang 2021-10-28 1934 PageWriteback(page))) { 4655e5e5f38726 Song Liu 2019-11-15 1935 /* 4655e5e5f38726 Song Liu 2019-11-15 1936 * khugepaged only works on read-only fd, so this 4655e5e5f38726 Song Liu 2019-11-15 1937 * page is dirty because it hasn't been flushed 4655e5e5f38726 Song Liu 2019-11-15 1938 * since first write. 4655e5e5f38726 Song Liu 2019-11-15 1939 */ 4655e5e5f38726 Song Liu 2019-11-15 1940 result = SCAN_FAIL; 4655e5e5f38726 Song Liu 2019-11-15 1941 goto out_unlock; 4655e5e5f38726 Song Liu 2019-11-15 1942 } 4655e5e5f38726 Song Liu 2019-11-15 1943 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1944 if (isolate_lru_page(page)) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1945 result = SCAN_DEL_PAGE_LRU; 042a30824871fa Hugh Dickins 2018-11-30 1946 goto out_unlock; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1947 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1948 99cb0dbd47a15d Song Liu 2019-09-23 1949 if (page_has_private(page) && 99cb0dbd47a15d Song Liu 2019-09-23 1950 !try_to_release_page(page, GFP_KERNEL)) { 99cb0dbd47a15d Song Liu 2019-09-23 1951 result = SCAN_PAGE_HAS_PRIVATE; 2f33a706027c94 Hugh Dickins 2020-05-27 1952 putback_lru_page(page); 99cb0dbd47a15d Song Liu 2019-09-23 1953 goto out_unlock; 99cb0dbd47a15d Song Liu 2019-09-23 1954 } 99cb0dbd47a15d Song Liu 2019-09-23 1955 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1956 if (page_mapped(page)) 869f7ee6f64773 Matthew Wilcox (Oracle 2022-02-15 1957) try_to_unmap(page_folio(page), 869f7ee6f64773 Matthew Wilcox (Oracle 2022-02-15 1958) TTU_IGNORE_MLOCK | TTU_BATCH_FLUSH); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1959 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1960 xas_lock_irq(&xas); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1961 xas_set(&xas, index); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1962 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1963 VM_BUG_ON_PAGE(page != xas_load(&xas), page); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1964 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1965 /* f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1966 * The page is expected to have page_count() == 3: f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1967 * - we hold a pin on it; 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1968 * - one reference from page cache; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1969 * - one from isolate_lru_page; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1970 */ f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1971 if (!page_ref_freeze(page, 3)) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1972 result = SCAN_PAGE_COUNT; 042a30824871fa Hugh Dickins 2018-11-30 1973 xas_unlock_irq(&xas); 042a30824871fa Hugh Dickins 2018-11-30 1974 putback_lru_page(page); 042a30824871fa Hugh Dickins 2018-11-30 1975 goto out_unlock; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1976 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1977 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1978 /* f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1979 * Add the page to the list to be able to undo the collapse if f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1980 * something go wrong. f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1981 */ f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1982 list_add_tail(&page->lru, &pagelist); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1983 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1984 /* Finally, replace with the new page. */ 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1985 xas_store(&xas, hpage); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1986 continue; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1987 out_unlock: f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1988 unlock_page(page); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1989 put_page(page); 042a30824871fa Hugh Dickins 2018-11-30 1990 goto xa_unlocked; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1991 } 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1992 nr = thp_nr_pages(hpage); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1993 180b0b616c523c Jiaqi Yan 2022-11-06 1994 if (!is_shmem) { 09d91cda0e8207 Song Liu 2019-09-23 1995 filemap_nr_thps_inc(mapping); eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1996 /* eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1997 * Paired with smp_mb() in do_dentry_open() to ensure eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1998 * i_writecount is up to date and the update to nr_thps is eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1999 * visible. Ensures the page cache will be truncated if the eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2000 * file is opened writable. eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2001 */ eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2002 smp_mb(); eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2003 if (inode_is_open_for_write(mapping->host)) { eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2004 result = SCAN_FAIL; eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2005 filemap_nr_thps_dec(mapping); eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2006 goto xa_locked; eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2007 } 09d91cda0e8207 Song Liu 2019-09-23 2008 } 042a30824871fa Hugh Dickins 2018-11-30 2009 xa_locked: 042a30824871fa Hugh Dickins 2018-11-30 2010 xas_unlock_irq(&xas); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2011 xa_unlocked: 042a30824871fa Hugh Dickins 2018-11-30 2012 6d9df8a5889c56 Hugh Dickins 2022-02-14 2013 /* 6d9df8a5889c56 Hugh Dickins 2022-02-14 2014 * If collapse is successful, flush must be done now before copying. 6d9df8a5889c56 Hugh Dickins 2022-02-14 2015 * If collapse is unsuccessful, does flush actually need to be done? 6d9df8a5889c56 Hugh Dickins 2022-02-14 2016 * Do it anyway, to clear the state. 6d9df8a5889c56 Hugh Dickins 2022-02-14 2017 */ 6d9df8a5889c56 Hugh Dickins 2022-02-14 2018 try_to_unmap_flush(); 6d9df8a5889c56 Hugh Dickins 2022-02-14 2019 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2020 if (result == SCAN_SUCCEED) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2021 /* 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2022 * Replacing old pages with new one has succeeded, now we 180b0b616c523c Jiaqi Yan 2022-11-06 2023 * attempt to copy the contents. f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2024 */ 2af8ff291848cc Hugh Dickins 2018-11-30 2025 index = start; 180b0b616c523c Jiaqi Yan 2022-11-06 @2026 list_for_each_entry(page, &pagelist, lru) { 2af8ff291848cc Hugh Dickins 2018-11-30 2027 while (index < page->index) { 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2028 clear_highpage(hpage + (index % HPAGE_PMD_NR)); 2af8ff291848cc Hugh Dickins 2018-11-30 2029 index++; 2af8ff291848cc Hugh Dickins 2018-11-30 2030 } 180b0b616c523c Jiaqi Yan 2022-11-06 2031 if (copy_highpage_mc(hpage + (page->index % HPAGE_PMD_NR), page)) { 180b0b616c523c Jiaqi Yan 2022-11-06 2032 result = SCAN_COPY_MC; 180b0b616c523c Jiaqi Yan 2022-11-06 2033 break; 180b0b616c523c Jiaqi Yan 2022-11-06 2034 } 180b0b616c523c Jiaqi Yan 2022-11-06 2035 index++; 180b0b616c523c Jiaqi Yan 2022-11-06 2036 } 180b0b616c523c Jiaqi Yan 2022-11-06 2037 while (result == SCAN_SUCCEED && index < end) { 180b0b616c523c Jiaqi Yan 2022-11-06 @2038 clear_highpage(hpage + (page->index % HPAGE_PMD_NR)); 180b0b616c523c Jiaqi Yan 2022-11-06 2039 index++; 180b0b616c523c Jiaqi Yan 2022-11-06 2040 } 180b0b616c523c Jiaqi Yan 2022-11-06 2041 } 180b0b616c523c Jiaqi Yan 2022-11-06 2042 180b0b616c523c Jiaqi Yan 2022-11-06 2043 if (result == SCAN_SUCCEED) { 180b0b616c523c Jiaqi Yan 2022-11-06 2044 /* 180b0b616c523c Jiaqi Yan 2022-11-06 2045 * Copying old pages to huge one has succeeded, now we 180b0b616c523c Jiaqi Yan 2022-11-06 2046 * need to free the old pages. 180b0b616c523c Jiaqi Yan 2022-11-06 2047 */ 180b0b616c523c Jiaqi Yan 2022-11-06 2048 list_for_each_entry_safe(page, tmp, &pagelist, lru) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2049 list_del(&page->lru); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2050 page->mapping = NULL; 042a30824871fa Hugh Dickins 2018-11-30 2051 page_ref_unfreeze(page, 1); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2052 ClearPageActive(page); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2053 ClearPageUnevictable(page); 042a30824871fa Hugh Dickins 2018-11-30 2054 unlock_page(page); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2055 put_page(page); 2af8ff291848cc Hugh Dickins 2018-11-30 2056 } 180b0b616c523c Jiaqi Yan 2022-11-06 2057 180b0b616c523c Jiaqi Yan 2022-11-06 2058 xas_lock_irq(&xas); 180b0b616c523c Jiaqi Yan 2022-11-06 2059 if (is_shmem) 180b0b616c523c Jiaqi Yan 2022-11-06 2060 __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); 180b0b616c523c Jiaqi Yan 2022-11-06 2061 else 180b0b616c523c Jiaqi Yan 2022-11-06 2062 __mod_lruvec_page_state(hpage, NR_FILE_THPS, nr); 180b0b616c523c Jiaqi Yan 2022-11-06 2063 180b0b616c523c Jiaqi Yan 2022-11-06 2064 if (nr_none) { 180b0b616c523c Jiaqi Yan 2022-11-06 2065 __mod_lruvec_page_state(hpage, NR_FILE_PAGES, nr_none); 180b0b616c523c Jiaqi Yan 2022-11-06 2066 /* nr_none is always 0 for non-shmem. */ 180b0b616c523c Jiaqi Yan 2022-11-06 2067 __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2068 } 180b0b616c523c Jiaqi Yan 2022-11-06 2069 /* Join all the small entries into a single multi-index entry. */ 180b0b616c523c Jiaqi Yan 2022-11-06 2070 xas_set_order(&xas, start, HPAGE_PMD_ORDER); 180b0b616c523c Jiaqi Yan 2022-11-06 2071 xas_store(&xas, hpage); 180b0b616c523c Jiaqi Yan 2022-11-06 2072 xas_unlock_irq(&xas); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2073 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2074 SetPageUptodate(hpage); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2075 page_ref_add(hpage, HPAGE_PMD_NR - 1); 6058eaec816f29 Johannes Weiner 2020-06-03 2076 if (is_shmem) 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2077 set_page_dirty(hpage); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2078 lru_cache_add(hpage); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2079 042a30824871fa Hugh Dickins 2018-11-30 2080 /* 042a30824871fa Hugh Dickins 2018-11-30 2081 * Remove pte page tables, so we can re-fault the page as huge. 042a30824871fa Hugh Dickins 2018-11-30 2082 */ 34488399fa08fa Zach O'Keefe 2022-09-22 2083 result = retract_page_tables(mapping, start, mm, addr, hpage, 34488399fa08fa Zach O'Keefe 2022-09-22 2084 cc); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2085 unlock_page(hpage); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2086 hpage = NULL; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2087 } else { 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2088 /* Something went wrong: roll back page cache changes */ 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2089 xas_lock_irq(&xas); 2f55f070e5b80f Miaohe Lin 2022-06-25 2090 if (nr_none) { aaa52e340073b7 Hugh Dickins 2018-11-30 2091 mapping->nrpages -= nr_none; aaa52e340073b7 Hugh Dickins 2018-11-30 2092 shmem_uncharge(mapping->host, nr_none); 2f55f070e5b80f Miaohe Lin 2022-06-25 2093 } aaa52e340073b7 Hugh Dickins 2018-11-30 2094 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2095 xas_set(&xas, start); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2096 xas_for_each(&xas, page, end - 1) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2097 page = list_first_entry_or_null(&pagelist, f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2098 struct page, lru); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2099 if (!page || xas.xa_index < page->index) { f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2100 if (!nr_none) f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2101 break; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2102 nr_none--; 59749e6ce53735 Johannes Weiner 2016-12-12 2103 /* Put holes back where they were */ 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2104 xas_store(&xas, NULL); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2105 continue; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2106 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2107 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2108 VM_BUG_ON_PAGE(page->index != xas.xa_index, page); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2109 f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2110 /* Unfreeze the page. */ f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2111 list_del(&page->lru); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2112 page_ref_unfreeze(page, 2); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2113 xas_store(&xas, page); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2114 xas_pause(&xas); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2115 xas_unlock_irq(&xas); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2116 unlock_page(page); 042a30824871fa Hugh Dickins 2018-11-30 2117 putback_lru_page(page); 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2118 xas_lock_irq(&xas); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2119 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2120 VM_BUG_ON(nr_none); 180b0b616c523c Jiaqi Yan 2022-11-06 2121 /* 180b0b616c523c Jiaqi Yan 2022-11-06 2122 * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. 180b0b616c523c Jiaqi Yan 2022-11-06 2123 * This undo is not needed unless failure is due to SCAN_COPY_MC. 180b0b616c523c Jiaqi Yan 2022-11-06 2124 */ 180b0b616c523c Jiaqi Yan 2022-11-06 2125 if (!is_shmem && result == SCAN_COPY_MC) 180b0b616c523c Jiaqi Yan 2022-11-06 2126 filemap_nr_thps_dec(mapping); 180b0b616c523c Jiaqi Yan 2022-11-06 2127 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2128 xas_unlock_irq(&xas); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2129 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2130 hpage->mapping = NULL; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2131 } 042a30824871fa Hugh Dickins 2018-11-30 2132 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2133 if (hpage) 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2134 unlock_page(hpage); f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2135 out: f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2136 VM_BUG_ON(!list_empty(&pagelist)); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2137 if (hpage) { 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2138 mem_cgroup_uncharge(page_folio(hpage)); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2139 put_page(hpage); c6a7f445a2727a Yang Shi 2022-07-06 2140 } 95c66f836a4276 Gautam Menghani 2022-10-26 2141 95c66f836a4276 Gautam Menghani 2022-10-26 2142 trace_mm_khugepaged_collapse_file(mm, hpage, index, is_shmem, addr, file, nr, result); 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2143 return result; f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2144 } f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2145 -- 0-DAY CI Kernel Test Service https://01.org/lkp